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Chapter  1 

GENERAL  GUIDANCE 

1.1.  The  Purpose  of  Test  and  Evaluation  (T&E).  The  primary  functions  of  T&E  are  to  deliver  effective  and  suitable 
systems,  and  to  identify  and  resolve  deficiencies  as  early  as  possible.  The  Air  Force  will  conduct  T&E  to  support  the 
acquisition,  modification,  upgrade,  and  sustainment  of  weapon  systems  and  product  groups.  The  single  manager  (SM)  will 
ensure  systems  and  product  groups  meet  Air  Force  peacetime  and  wartime  missions  by: 

•  Providing  timely,  accurate,  and  affordable  information  to  decision  makers  so  they  may  assess  whether  a  system  or 
technique  is  useful  and  cost-effective  to  the  military. 

•  Identifying  and  resolving  deficiencies  early. 

•  Supporting  decision  makers  in  assessing  how  to  best  employ  limited  resources. 

•  Reducing  risks  during  the  acquisition  and  sustainment  phases  of  systems’  life  cycles. 

•  Ensuring  the  acquisition  community  delivers  operationally  effective  and  suitable  systems  to  Air  Force  users. 

•  Ensuring  systems  continue  to  be  operationally  effective  and  suitable  (sustained)  throughout  their  life  cycles. 
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•  Giving  operational  users  the  information  needed  to  develop  tactics,  doctrines,  procedures,  and  system  enhancements. 

1.2.  The  T&E  Process.  The  T&E  process  is  based  on  the  scientific  method  and  the  principle  of  “predict  -  test  -  compare” 
as  shown  in  figure  1.1.  The  Air  Force  acquisition  and  test  communities  will  apply  the  T&E  process  throughout  the  system 
acquisition  cycle  to  identify  and  resolve  deficiencies,  and  demonstrate  system  effectiveness  and  suitability.  The  core  elements 
of  the  T&E  process  are  described  below: 

1.2.1.  Scientific  Method.  The  T&E  process  uses  the  scientific  method  to  build  experiments  or  tests  as  follows:  a  hypothesis 
is  developed  as  a  prediction  of  performance;  the  experiment  or  test  is  executed;  and  the  data  analyzed  and  compared  to  the 
original  hypothesis  or  prediction.  If  the  identified  risk  is  acceptable,  the  experiment  or  test  is  complete.  If  the  risk  is  not 
acceptable,  refinements  are  made  to  one  or  more  of  the  following:  the  system’s  design;  the  system’s  operating  concept;  the 
experiment’s  design  (e.g.,  test  method,  test  instrumentation,  or  data  analysis  method);  or  the  original  prediction.  The  test  is 
then  repeated.  Air  Force  acquisition  and  test  personnel  will  use  the  T&E  process  to  identify  and  resolve  problems  early, 
decrease  acquisition  program  risk,  and  increase  decision  maker  confidence  in  T&E  results. 

1.2.2.  Predict.  This  part  of  the  T&E  process  is  supported  by  modeling  and  simulation  (M&S)  tools  and  the  results  of  previous 
testing.  M&S  tools  help  develop  T&E  concepts  and  predict  results  prior  to  testing.  They  can  predict  and  explore  potential 
problems  for  early  resolution,  and  reduce  the  amount  of  actual  testing  needed.  Typically  during  this  step,  a  model  of  the  system 
under  test  interacts  with  environmental,  threat,  and  other  weapon  system  models,  with  various  levels  of  fidelity,  to  estimate 
and  predict  system  performance. 

1.2.3.  Test.  Testing  obtains  or  verifies  data  for  determining  the  degree  to  which  systems  meet  stated  requirements.  This  part 
of  the  T&E  process  is  supported  by  test  facilities  and  range  instrumentation.  These  devices  are  used  to  produce  and  record 
data  from  testing. 

1.2.4.  Compare.  This  part  of  the  T&E  process  compares  measured  test  data  to  predicted  results,  followed  by  evaluation  of  the 
differences  found.  As  these  differences  are  reduced,  confidence  is  gained  about  the  design  and  performance  of  the  system 
under  test.  This  part  of  the  T&E  process  is  supported  by  requirements  documents,  test  plans,  test  results,  analytical  and 
statistical  tools,  evaluations,  and  summaries  from  current  and  previous  test  projects.  Test  reports  will  be  concise,  timely,  and 
contain  critical  information  needed  by  decision  makers. 

Figure  1.1.  The  Scientific  Method. 


NOTE:  Early  problem  identification  and  resolution  must  take  place  at  all  stages  of  development. 

1.3.  Test  Capabilities  and  Facilities.  The  capabilities,  facilities,  and  instrumentation  used  in  the  T&E  process  are  shown 
in  figure  1.2.  The  system  design  begins  with  computer  models  and  simulations  capable  of  numerous  low  cost  replications, 
progresses  through  integrated  facilities  of  higher  cost  and  complexity,  but  with  fewer  replications,  and  finishes  with  highly 
selective  testing  in  realistic,  open-air  environments.  Each  step  predicts  results  for  the  next  step  in  the  system  design,  and 
reduces  unnecessary  and  increasingly  costly  tests  later  in  the  program. 

1.3.1.  Modeling  and  Simulation  (M&S).  The  T&E  process  uses  computer-aided  simulations  and  analysis  prior  to  testing  to 
help  design  tests  and  predict  test  results,  and  after  testing  to  extrapolate  test  results  to  other  conditions.  M&S  will  be  used  to 
provide  constant  feedback  for  system  development  and  improvements.  Use  of  M&S  should  begin  during  the  Concept 
Exploration  Phase  of  the  Acquisition  Process  and  continue  throughout  the  system’s  life-cycle.  Use  of  M&S  should  be 
consistent  with  Air  Force  approved  infrastructure  initiatives  and  system  architectures  such  as  J-MASS,  JSIMS,  and  JWARS. 
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1 .3.2.  Data  Measurement  Facility.  Measurement  facilities  quantify  the  capabilities  and  limitations  of  systems,  subsystems,  and 
technologies.  They  provide  needed  inputs  to  computer  simulations,  explore  and  evaluate  advanced  technologies,  confirm 
design  capabilities,  identify  design  problems,  and  determine  employment  options. 

1.3.3.  System  Integration  Laboratory  (SIL).  SILs  are  facilities  designed  to  test  the  interoperability  and  compatibility  of 
components,  subsystems,  and  systems  when  they  are  integrated  with  other  systems  or  functions.  They  are  used  to  evaluate 
individual  hardware  and  software  interactions  and,  at  times,  involve  the  entire  weapon  system.  A  variety  of  computer 
simulations  and  test  equipment  are  used  to  generate  scenarios  and  environments  to  test  for  functional  performance,  reliability, 
and  safety. 

Figure  1,2.  Test  and  Evaluation  Facilities  and  Capabilities  Hierarchy. 


MODELING  AND  SIMULATION 


TIME 


1.3.4.  Hardware-in-the-Loop  (HITL).  HITL  facilities  include  specialized  indoor  facilities  providing  a  secure  environment  to 
test  techniques  and  hardware  against  simulations  of  hostile  weapon  system  hardware  or  actual  hostile  weapon  system 
hardware.  They  are  used  to  determine  threat  system  susceptibility  and  effectiveness,  and  to  evaluate  the  performance  of 
systems  and  techniques.  HITL  is  the  first  opportunity  to  test  uninstalled  system  components  in  a  realistic  simulated 
environment. 

1.3.5.  Installed  System  Test  Facility  (ISTF).  ISTFs  provide  a  secure  capability  to  evaluate  systems  installed  on,  or  integrated 
with,  host  platforms.  The  system  under  test  is  stimulated  by  threat  signal  generators  and  its  responses  evaluated  to  provide 
critical,  integrated  system  performance  information.  Their  primary  purpose  is  to  evaluate  integrated  systems  in  installed 
configurations  to  test  specific  functions  of  complete,  full-scale  weapon  systems. 

1.3.6.  Open-Air  Range  (OAR).  OARs  are  used  to  evaluate  systems  in  realistic  backgrounds,  clutter,  noise,  and  dynamic 
environments.  These  resources  are  typically  divided  into  subcategories  of  test  ranges  and  airborne  testbeds.  The  primary 
purpose  of  open-air  testing  is  to  evaluate  systems  in  real-world,  representative  environments  and  operating  conditions. 
Open-air  testing  validates  system  performance  and  effectiveness  to  a  high  confidence  level.  If  properly  structured,  open-air 
testing  can  be  used  to  validate  and  calibrate  ground  test  facilities  and  models. 
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1.4.  lypes  of  T&E.  There  are  two  major  types  of  Tc&E  in  the  defense  system  acquisition  process:  developmental  test  and 
evaluation  (DT&E);  and  operational  test  and  evaluation  (OT&E).  Government  agencies  and  contractors  may  conduct  DT&E. 
DT&E  and  OT&E  will  be  combined  when  appropriate;  however,  a  dedicated  phase  of  OT&E  is  normally  required.  OT&E  is 
described  in  AFI  99-102. 


Chapter  2 

DEVELOPMENTAL  TEST  AND  EVALUATION 


Section  A — General  Information 

2.1.  Purpose  of  Developmental  Test  and  Evaluation  (DT&E).  Various  kinds  of  DT&E  are  conducted  throughout  a 
system’s  life  cycle  to  ensure  the  Air  Force  acquires  and  maintains  operationally  effective  and  suitable  systems  which  meet 
users’  needs.  Both  contractor  and  government  personnel  conduct  DT&E  to  identify  and  resolve  deficiencies  as  early  as 
possible.  Decision  makers  use  DT&E  results  to  verily  the  extent  to  which  design  risks  have  been  minimized,  verify  contract 
performance,  determine  system  safety,  assess  military  utility  and  system  reliability,  and  determine  system  readiness  for 
dedicated  OT&E.  On  fielded  systems,  DT&E  is  used  to  extend  the  useful  military  life  of  systems,  increase  current 
performance  capabilities,  analyze  materiel  deficiencies,  and  implement  solutions.  The  early  identification  and  resolution  of 
deficiencies  prevents  problems  from  continuing  through  the  balance  of  testing  and  into  the  field. 

2.2.  Scope  of  Developmental  Test  and  Evaluation  (DT&E).  This  AFI  uses  the  general  term  “DT&E”  to  cover  a  broad 
range  of  T&E  activities  conducted  during  all  phases  of  a  system’s  life  cycle.  DT&E  supports  virtually  every  new  or  fielded 
system  and/or  component  at  various  points  in  its  life  cycle.  The  single  manager  (SM)  or  item  manager  proposes  the  type  of 
funding  required  for  each  development  program  depending  on  the  nature  and  objectives  of  the  work  required.  The  test  process 
guidance  in  this  AFI  will  be  used  to  conduct  DT&E  during  each  phase  of  a  system’s  life  cycle,  as  appropriate. 

2.2.1.  Developmental  Test  and  Evaluation  (DT&E)  During  Acquisition.  DT&E  conducted  during  acquisition  provides 
complete  and  reliable  data  for  estimating  the  military  utility  of  new  systems  or  items,  and  forms  the  basis  for  making  decisions 
to  continue  the  acquisition  process.  In  some  instances,  DT&E  may  continue  past  Milestone  III  for  highly  concurrent  programs 
where  development  and  production  overlap.  DT&E  is  conducted  on  new  systems,  or  on  product  improvements  to  fielded 
systems,  to  expand  their  current  performance  envelopes  or  capabilities.  In  addition,  DT&E  during  acquisition: 

•  Evaluates  design  approaches,  validates  analytical  models,  quantifies  contract  technical  performance  and  manufacturing 
quality,  measures  progress  in  system  engineering  design  and  development,  minimizes  design  risks,  and  predicts 
integrated  system  performance  (effectiveness  and  suitability)  in  operationally  relevant  environments. 

•  Identifies,  tracks,  and  resolves  design  deficiencies  as  early  as  possible.  Also  identifies  enhancements. 

•  Provides  data  for  cost-performance  trade  decisions  before  system  downselect,  and  during  system  development. 

•  Is  funded  with  research,  development,  test  and  evaluation  (RDT&E)  (3600)  funds. 

•  Supports  the  acquisition  of  new  or  improved  materiel  or  operational  capabilities  before  acceptance  for  operational  use 
or  production. 

•  Supports  the  decision  to  certify  the  system  ready  for  dedicated  Initial  Operational  Test  and  Evaluation  (lOT&E). 

2.2.2.  Developmental  Test  and  Evaluation  (DT&E)  During  Sustainment.  DT&E  is  conducted  to  support  the  sustainment 
portion  (generally  after  Milestone  III)  of  a  system’s  life  cycle.  This  DT&E  includes  all  T&E  activities  needed  to  keep  systems 
current  or  extend  their  useful  life  after  production  and  fielding.  DT&E  for  sustainment  may  also  include  efforts  to  expand  or 
upgrade  the  current  performance  envelopes  and  capabilities  of  systems.  DT&E  for  sustainment  will  follow  the  test  process 
guidance  in  this  AFI.  In  addition,  DT&E  during  sustainment: 

•  Is  planned,  conducted,  and  reported  for  improvements,  modifications,  or  upgrades  to  fielded  systems  which: 

•  Extend  their  useful  military  life  within  the  current  performance  envelope  or  capabilities.  In  these  cases,  use 
Procurement  (e.g.,  3010, 3020,  3080)  funds  for  systems  still  in  production,  or  O&M  (3400)  funds  for  systems  no  longer 
in  production  but  still  in  the  operational  inventory, 

•  Expand  or  increase  the  current  performance  envelopes  or  capabilities.  In  these  cases,  use  RDT&E  (3600)  funds. 

•  Identifies,  tracks,  and  resolves  system  deficiencies  as  early  as  possible.  Identifies  enhancements.  Also  identifies 
quality,  reliability,  maintainability,  and  safety  problems  and  solutions. 

•  Ensures  fielded  systems  continue  to  perform  as  required. 

•  Provides  information  which  may  lead  to  future  modification  or  upgrade  programs. 

•  Supports  modification  or  upgrade  programs  for  fielded  systems  when  a  sufficient  amount  of  integration  testing  is 
needed. 
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•  Provides  aging  and  surveillance  testing. 

•  Verifies  changes  in  operational  environments. 

2.2.3.  Qualification  Test  and  Evaluation  (QT&E).  QT&E  is  a  modified  form  of  DT&E  conducted  on  commercial  off-the-shelf 
(COTS),  nondevelopmental  items  (NDI),  and  government  furnished  equipment  (GFE).  Candidate  systems  for  QT&E  require 
little  or  no  government  funded  research  and  development  (R&D),  engineering,  design,  or  integration  efforts.  In  addition, 
QT&E: 

•  Is  planned,  conducted,  and  reported  following  the  same  test  process  guidance  in  this  AFI  applicable  to  all  DT&E. 

•  Identifies,  tracks,  and  resolves  system  deficiencies  as  early  as  possible.  Also  identifies  enhancements. 

•  Supports  the  verification  of  changes,  modifications,  or  upgrades  to  fielded  systems. 

•  Is  funded  with  Operations  and  Maintenance  (O&M)  (3400)  funds,  or  Procurement  (e.g.,  3010,  3020,  3080)  funds. 

•  Supports  the  decision  to  certify  the  system  ready  for  dedicated  Qualification  Operational  Test  and  Evaluation 
(QOT&E). 

2.2.4.  Compatibility,  Interoperability,  and  Integration  Testing.  A  type  of  qualification  test  and  evaluation  (QT&E)  that  ensures 
users’  compatibility,  interoper-ability,  and  integration  requirements  are  evaluated  in  operationally  relevant  environments.  It 
demonstrates  correct  system  interaction  and  responsiveness,  and  may  continue  throughout  the  system’s  life  cycle.  For 
compatibility,  interoperability,  and  integration  testing  pertaining  to  network  software  and  hardware  components,  see  AFI 
33-108,  Compatibility,  Interoperability,  and  Integration  of  Command,  Control,  Communications,  Computer  (C4)  Systems. 

2.2.5.  Contractor  DT&E.  Contractors  will  conduct  DT&E  in  the  acquisition  strategy.  Similar  to  government  conducted 
DT&E,  contractor  DT&E  and  QT&E  identify,  track,  and  resolve  system  deficiencies  as  early  as  possible.  The  SM  will  oversee 
all  contractor  conducted  T&E,  and  retain  final  approval  authority  over  all  contractor  test  plans. 

2.2.5. 1.  Government  Oversight.  The  greater  the  reliance  on  contractors  for  testing,  the  greater  the  need  for  oversight  by 
knowledgeable  government  officials.  The  obligation  of  government  funds,  the  final  evaluation  of  results,  and  the  approval  of 
test  reports  will  remain  government  responsibilities.  See  0MB  Policy  Letter  92-1,  Inherently  Governmental  Functions,  for 
guidance  about  which  functions  must  be  retained  by  the  government. 

2.2. 5. 2.  Contractor  Responsibilities.  Contractor  DT&E  responsibilities  must  be  clearly  defined  in  the  request  for  proposal 
(RFP)  and  the  test  and  evaluation  master  plan  (TEMP),  to  include  the  requirements  for  independent  and  joint  analysis  and 
reporting  with  the  contractor.  An  approved  deficiency  reporting  (DR)  system  compatible  with  the  government’s  DR  system 
must  be  used  (see  paragraph  2.18).  The  SM,  with  the  support  of  the  RTO,  will  approve  all  contractor  test  plans  and  reports, 
and  will  oversee  contractor  testing.  Contractors  will  not  be  designated  as  the  responsible  test  organization  (RTO),  but  test  and 
evaluation  tasks  may  be  delegated  to  them. 

2.2.6.  Live  Fire  Test  and  Evaluation  (LFT&E).  LFT&E  provides  an  assessment  of  the  vulnerability  (or  lethality)  of  a  weapon 
system  as  it  progresses  through  its  development  and  prior  to  the  full-rate  production  decision  (Milestone  III).  If  the  system 
under  development  is  a  covered  system  for  LFT&E  purposes,  the  SM  will  ensure  LFT&E  is  funded  and  the  test  plan  approved 
before  the  MS  II  decision.  LFT&E  must  be  completed  before  the  Milestone  III  decision.  See  AFI  99-105,  Live  Fire  Test  and 
Evaluation,  and  DoD  5000.2-R,  Appendix  IV. 

2.2.7.  Production  Acceptance  Test  (PAT).  The  SM  or  contractor,  as  appropriate,  will  endure  DT&E  is  conducted  on  production 
items  to  demonstrate  that  specifications  and  performance-based  requirements  of  the  procuring  contracts  or  agreements  have 
been  fulfilled.  The  government  will  oversee  any  contractor  conducted  PATs.  The  following  types  of  PAT  are  further  defined 
in  Appendix  III: 

•  First  article  test  (FAT). 

•  Lot  acceptance  test  (LAT). 

•  Preproduction  qualification  test  (PPQT).  SMs  must  report  PPQT  results  to  decision  makers  prior  to  the  applicable 
production  decision. 

•  Production  qualification  test  (PQT).  SMs  must  report  PQT  results  to  decision  makers  prior  to  the  full-rate  production 
decision. 

2.2.8.  Combined  Test  and  Evaluation  (T&E).  SMs  will  structure  T&E  programs  to  integrate  all  developmental,  operational, 
live  fire  test,  and  modeling  and  simulation  (M&S)  activities  conducted  by  different  agencies  into  an  efficient  continuum.  SMs 
and  testers  will  use  a  combined  test  approach  when  there  are  test  efficiencies,  and  cost  and  schedule  savings.  For  example, 
DT&E  and  OT&E  may  be  combined  when  the  data,  test  scenarios,  measures  of  effectiveness,  resources,  and  objectives  of  both 
tests  are  similar  and  compatible.  The  SM  and  RTO  may  form  teaming  arrangements  with  the  contractor  if  that  is  an  effective 
way  to  reduce  total  cost  and  duplications  in  data  and  resources.  However,  a  combined  T&E  approach  must  not  compromise 
either  developmental  or  operational  test  objectives.  See  paragraph  6.10  for  more  details. 

2.3,  Examples  of  Programs  and  Activities  Covered  by  This  Instruction. 

•  PMD-directed  programs  in  acquisition  categories  (ACAT)  I  through  III. 

•  Modifications  and  upgrades. 
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•  COTS,  NDI,  and  GFE  systems. 

•  Automated  information  systems  (AIS),  automated  data  processing  (ADP)  systems,  and  network  software  and  hardware 
components. 

•  Programs  that  develop  or  procure  one  or  only  a  few  units  over  an  extended  period  such  as  a  satellite  system. 

•  Prototype  systems. 

•  Aircraft/stores  certification  (SEEK  EAGLE)  (AFI  63-104). 

•  Programs  directed  by  higher  authority  such  as  the  Secretary  of  the  Air  Force  (SAF)  or  Chief  of  Staff  of  the  Air  Force 
(CSAF). 

•  Systems  developed  using  evolutionary  or  incremental  acquisition  strategies. 

•  Systems  produced  for  foreign  military  sales. 

•  Advanced  concept  technology  demonstrations  (ACTD). 

•  Technology  demonstrations  not  covered  by  AFPD  61-1. 

•  Preplanned  product  improvements  (P3I). 

•  Hardware  and  software  integration. 

•  Production  acceptance  test  (PAT). 

•  Live  fire  test  and  evaluation  (LFT&E)  programs. 

•  Logistics  test  and  evaluation. 

2.4.  Examples  of  Programs  and  Activities  Not  Covered  by  This  Instruction. 

•  Nuclear  components  governed  by  joint  DoD-Department  of  Energy  agreements. 

•  Threat  simulators  evaluated  under  the  DoD  threat  simulator  program  and  Air  Force  simulator  validation  process. 

•  Training  systems  that  are  totally  contractor  provided,  maintained,  and  operated. 

•  Science  and  technology  (S&T)  programs  falling  under  the  guidance  of  AFPD  61-1  and  accompanying  AFIs. 

•  Industrial  maintenance  inspections  (IMI). 

•  OT&E  programs  covered  by  AFI  99-102. 

2.5.  Funding  for  DT&E.  Explicit  guidance  on  when  to  use  RDT&E,  O&M,  or  Procure-ment  funding  for  DT&E  is 
contained  in  DoD  7000. 14-R,  Vol  2A,  Financial  Management  Regulation,  Chapter  1.  The  term  “RDT&E”  generally  refers  to 
the  type  of  funding  appropriation  (3600)  intended  for  research,  development,  test  and  evaluation  efforts.  A  key  decision  criteria 
is  whether  or  not  the  proposed  DT&E  supports  a  program  that  expands  the  current  performance  envelope  or  capabilities  of 
the  system.  When  there  is  doubt  as  to  the  proper  assignment  of  T&E  costs  between  appropriations,  seek  additional  guidance 
through  financial  management  channels,  or  through  HQ  USAF/TER  and  SAF/AQXR.  Also  see  AFI  65-601,  Volume  1,  Budget 
Guidance  and  Procedures. 

2.6.  Requests  to  Modify  Requirements  for  DT&E.  HQ  USAF/TE  is  the  sole  waiver  authority  for  this  instruction. 
Requests  for  waiver  must  be  submitted  with  rationale  and  user,  developer,  and  tester  concurrence  in  writing  through 
AFMC/DO  to  HQ  USAF/TE.  The  SM  will  document  any  approved  waivers  to  DT&E  procedures  or  requirements  in  the 
TEMP,  and  the  program  element  monitor  (PEM)  will  document  waivers  in  the  PMD. 

Section  B — Other  Types  of  T&E 

in.  Operational  Test  and  Evaluation  (OT&E).  OT&E  is  conducted  in  an  environment  as  realistic  as  possible  to 
determine  the  system’s  operational  effectiveness  and  operational  suitability,  and  to  ensure  operational  requirements  are  met. 
Initial  OT&E  (lOT&E)  supports  the  full-rate  production  or  fielding  decision,  and  follow-on  OT&E  (FOT&E)  is  conducted 
during  production  and  deployment  of  the  system.  For  more  information  about  OT&E,  see  AFI  99-102,  Operational  Test  and 
Evaluation. 

2.8.  Multiservice  T&E.  Multiservice  T&E  is  conducted  on  acquisition  or  sustainment  items  by  two  or  more  Services  and 
jointly  implemented.  The  designated  lead  Service  has  the  overall  responsibility  for  management  of  the  multiservice  program 
and  will  ensure  supporting  Service  requirements  are  included.  This  guidance  also  applies  to  testing  with  other  DoD  or  Federal 
agencies.  Multiservice  T&E  will  be  documented  in  the  PMD  and  the  TEMP. 

2.8.1.  Lead  Service.  When  the  Air  Force  is  designated  the  lead  Service  for  multiservice  T&E  the  SM  will  manage  the  program 
and  include  the  supporting  Service’s  requirements.  The  RTO  will  plan,  conduct,  and  report  the  DT&E  according  to  this 
instruction;  prepare  and  coordinate  a  single  DT&E  plan  and  a  final  report  summarizing  the  conclusions  and  recommendations 
of  each  Services’  reports;  and  explain  any  significant  differences  between  the  Services’  reports.  The  SM  will  harmonize  the 
DT&E  requirements  and  methods  of  other  Services  through  MOUs,  MOAs,  or  waivers  as  required  to  eliminate  conflicts  or 
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duplication  in  testing.  When  the  Air  Force  is  not  the  lead  Service,  the  SM  will  defer  to  the  other  Service’s  test  instructions, 
or  as  agreed. 

2.8.2.  Unique  Service  Requirements.  If  a  supporting  Service  has  unique  DT&E  objectives  or  operational  requirements,  that 
Service  will  plan,  fund,  conduct,  and  report  testing  for  those  unique  areas  following  their  own  Service  instructions,  or  as 
agreed.  All  supporting  Service  testing  must  be  documented  in  the  TEMR 

2.9.  Joint  Test  and  Evaluation  (JT&E).  JT&E  is  non-acquisition  testing  and  is  sponsored  by  the  Office  of  the  Secretary 
of  Defense  (OSD).  In  JT&E,  two  or  more  Services  cooperate  to  provide  legacy  products  and  data  of  interest  to  operational 
commanders.  See  AFI  99-106,  Joint  Test  and  Evaluationy  for  further  guidance. 

2.10.  Foreign  Comparative  Testing  (FCT).  FCT  is  a  DoD  test  and  evaluation  program  prescribed  by  Title  10  Section 
2350a(g)  and  centrally  managed  by  OSD/DTSE&E.  FCT  provides  funding  for  U.S.  test  and  evaluation  of  selected  equipment 
items  and  technologies  developed  by  allied  countries  when  these  systems  are  identified  as  having  good  potential  to  satisfy 
valid  DoD  requirements  (see  AFI  61-302,  Comparative  Research  and  Development  Agreements), 

2.11.  Advanced  Concept  Technology  Demonstrations  (ACTD).  ACTDs  are  a  means  of  demonstrating  the  use  of  mature 
technologies  to  meet  potential  military  needs.  They  provide  a  mechanism  for  rapid  transition  of  emerging  technologies  to 
operational  users.  ACTDs  are  not  a  formal  part  of  the  acquisition  process  or  formal  test  efforts,  but  “demonstrations”  under 
the  direction  of  an  ACTD  sponsor.  The  demonstration  is  jointly  sponsored  The  SM,  RTO,  and  OTA  will  participate  as  required 
by  the  ACTD  sponsor.  Headquarters  Air  Force,  and  Deputy  Under  Secretary  of  Defense  for  Advanced  Technology 
(DUSD/AT). 

Section  C — Testing  of  Commercial  and  Nondevelopmental  Items 

2.12.  Commercial-off-the-Shelf  (COTS)  Items,  COTS  items  include  already  developed  materiel,  systems,  subsystems, 
software,  and  hardware.  SMs  will  manage  DT&E  (normally  QT&E)  on  COTS  items  to  demonstrate  the  item,  modified  or  as 
is,  integrates  with  other  systems,  satisfies  the  mission  need,  and  meets  operational  requirements.  The  SM  will  ensure  the  test 
community,  including  the  OTA,  is  involved  in  the  COTS  acquisition  process  from  the  start.  The  SM  will  write  requests  for 
proposal  (RFP)  in  terms  that  enable  and  encourage  offerors  to  supply  commercial  items  to  meet  user  requirements.  The  test 
approach  will  be  tailored  to  the  type  of  system,  its  intended  operational  use,  and  the  amount  of  test  data  already  available. 
Sources  of  this  data  include: 

•  Contractor  test  results. 

•  Usage  data  from  other  military  and/or  commercial  customers, 

•  Results  from  independent  test  organizations. 

2.13.  Nondevelopmental  Items  (NDI).  NDIs  are  essentially  the  same  as  COTS  items.  The  procedures  used  to  test  NDIs 
are  the  same  as  for  COTS  programs. 

Section  D — General  DT&E  Guidance. 

2.14.  Starting  Early.  DT&E  and  OT&E  personnel  will  be  involved  as  early  as  possible  in  the  system  acquisition,  and  the 
problem  identification  and  resolution  processes.  The  Single-Face-to-Customer  (SFTC)  offices  will  be  involved  in  early  test 
planning  since  they  are  fundamental  to  the  success  of  the  overall  T&E  effort.  For  new  programs,  the  SPO  (or  other  acquisition 
agency  if  a  SPO  does  not  exist)  will  contact  an  SFTC  office  for  test  planning  assistance. 

2.15.  The  Single-Face-to-Customer  (SFTC)  Office.  From  a  new  program’s  inception,  the  SFTC  office  will  serve  as 
consultants  for  developing  a  complete  understanding  of  system  test  requirements.  They  will  participate  in  the  Air  Force  test 
process,  help  develop  test  documentation,  and  provide  T&E  options  and  associated  risks  to  the  RTO,  SPO,  and  OTA.  After 
AFMC  designates  or  approves  an  RTO  for  the  new  program,  the  RTO  becomes  the  single  T&E  point  of  contact  for  the  new 
program.  The  RTO  will  work  with  the  SPO  (or  other  acquisition  agency)  to  maintain  a  “single  face”  to  those  outside  the  DT&E 
community. 

2.16.  Assessing  the  Design.  An  integrated  product  team  (IPT)  composed  of  design,  system,  test,  manufacturing,  logistics, 
and  user  personnel  will  define  test  objectives,  methodologies,  and  evaluation  criteria  during  the  initial  phase  of  the  program 
(Phase  0).  These  elements  are  combined  to  form  the  overall  T&E  strategy  for  the  acquisition  and  sustainment  phases,  and 
guide  the  detailed  T&E  planning,  preparation,  and  execution  required  throughout  the  system  life  cycle.  Team  members  will 
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assess  design  deficiencies  using  technical  performance  measurement  techniques  from  the  System  Engineering  and 
Configuration  Management  Processes. 

2.17.  Resource  Planning.  DT&E  and  OT&E  personnel  will  together  determine  the  required  numbers  of  test  articles  and 
special  instrumentation  or  data  requirements;  simulators  or  surrogates  of  threat  weapons  or  devices;  and  other  test  resources. 
The  SFTCs  will  ensure  customers  are  made  aware  of  all  applicable  test  resources  in  time  to  include  the  use  of  those  resources 
in  their  test  plan.  AFI  99-109,  Test  Resource  Planning,  explains  the  test  resource  planning  process. 

2.18.  Establishing  a  Deficiency  Reporting  (DR)  System.  At  the  beginning  of  each  acquisition  program,  the  SM  will 
establish  a  DR  system.  TO  00-35D-54,  USAF  Deficiency  Reporting  and  Investigation  System,  describes  procedures  for 
identifying,  reporting,  tracking,  and  resolving  system  deficiencies  identified  during  T&E.  During  DT&E  execution,  the 
contractor  and  RTO  will  identify  and  document  all  deficiencies  as  early  as  possible,  and  recommend  corrective  actions 
according  to  the  procedures  established  for  that  program. 

2.18.1.  Use  of  Contractor-Based  Deficiency  Report  System.  The  SM  must  include  in  the  RFP  the  requirement  for  the 
contractor  to  establish  a  DR  system  for  contractor  conducted  tests.  The  contractor’s  system  must  be  compatible  with,  and 
provide  complete  and  useable  results  to,  the  government’s  DR  process.  The  prime  contractor  will  be  required  to  enforce  these 
DR  requirements  for  its  subcontractors  and  suppliers.  The  government  will  have  access  to  the  contractor’s  DR  system  and  be 
able  to  transfer  relevant  information  to  the  government  DR  process. 

2.18.2.  Setting  Priorities.  The  RTO,  OTA,  SM,  and  users  will  prioritize  all  open  DRs  for  any  Air  Force  programs  involving 
DT&E  and  OT&E.  If  the  SM  cannot  correct  or  resolve  known  DRs  before  dedicated  OT&E  begins,  or  defers  fixes  for  any 
DRs  past  OT&E,  the  OTA  and  users  will  prioritize  and  the  SM  will  analyze  the  operational  impacts  of  those  deficiencies  and 
limitations.  The  SM  and  OTA  will  reach  agreement  during  the  certification  process  for  dedicated  lOT&E,  and  develop  a  plan 
for  resolving  and  retesting  deficiencies  found  in  operational  testing. 

2. 1 8.3.  Waivers.  A  waiver  must  be  attained  from  AFMC/ENPP  if  the  government  and/or  contractor  uses  a  DR  system  different 
than  that  described  in  TO  00-35D-54. 

2.19.  Operationally  Relevant  Testing.  IPT  personnel  must  plan  to  conduct  government  DT&E  in  as  operationally  relevant 
an  environment  as  possible  without  compromising  the  goals  of  DT&E.  This  focus  on  test  realism  in  both  planning  and 
execution  is  important  for  identifying  and  resolving  deficiencies  earlier.  Once  a  system  has  reached  OT&E,  deficiencies  are 
more  difficult  and  costly  to  resolve.  Developmental  testers  will  evaluate  the  system  against  user  requirements  (as  stated  in  the 
ORD)  as  well  as  the  system  specifications.  Operational  suitability  will  be  given  equal  consideration  with  operational 
effectiveness.  Operationally  relevant  testing  will  be  described  in  the  TEMP  and  in  detailed  test  plans  (see  AFPAM  99-116,  Test 
and  Evaluation  Management  Guide). 

2.20.  Using  Event-Driven  Schedules  and  Exit  Criteria.  DoDD  5000.1  emphasizes  the  importance  of  event-driven 
schedules  (vice  calendar-driven)  and  exit  criteria  throughout  the  acquisition  process.  The  SM  will  ensure  the  planned  DT&E 
is  sufficient  to  verify  a  program  meets  the  test  objectives  and  exit  criteria  for  each  acquisition  phase.  The  SM  and  RTO  must 
ensure  adequate  time  is  scheduled  between  DT&E  events,  and  before  commencement  of  dedicated  OT&E.  The  SM  and  RTO 
will  not  consider  any  acquisition  phase  complete  unless  all  significant  test  events  and  exit  criteria  are  successfully  completed, 
and  all  significant  deficiencies  have  been  resolved  before  certification  of  readiness  for  dedicated  OT&E. 

2.21.  Ensuring  Environmental  Compliance  and  Pollution  Prevention  During  T&E.  SMs  and  RTOs  must  comply  with 
all  Federal,  state,  and  local  environmental  standards.  The  SM  must  ensure  the  Environmental  Impact  Analysis  Process,  to 
include  Environmental  Assessments  and  Federal  Consistency  Determinations,  are  integrated  into  the  DT&E  planning  process. 
The  SM  will  evaluate  occupational  health  and  hazardous  waste  issues  during  DT&E  to  minimize  adverse  effects  on 
supportability. 

2.22.  Disclosing  Test  Data  to  Foreign  Nationals.  The  test  community  must  use  AFI  16-201,  Disclosure  of  Military 
Information  to  Foreign  Governments  and  International  Organizations,  to  determine  when  to  disclose  test  data  or  materials  to 
foreign  nationals.  AFI  16-201  also  governs  the  disclosure  of  foreign  nationals’  access  to  Air  Force  equipment,  facilities,  and 
aircraft. 

2.23.  Use  of  Government  Test  Facilities.  The  SFTC  offices  and  test  centers  will  provide  all  potential  users  with 
information  and  assistance  on  using  the  Major  Range  and  Test  Facility  Base  (MRTFB)  and  other  government  test  facilities. 
The  SFTCs  will  help  develop  test  support  requirements  for  users  upon  request.  Use  of  the  MRTFB  is  described  in  DoDD 
3200.11,  The  Major  Range  and  Test  Facility  Base. 
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2.23.1.  Air  Force  Use  of  Test  Facilities.  SMs  and  test  planners  will  plan  to  use  Air  Force  test  facilities  first,  followed  by  DoD 
(MRTFB)  facilities,  to  the  maximum  extent  possible  within  acquisition  program  cost  and  schedule  constraints. 
Contractor-owned  facilities  may  be  considered  when  DoD  facilities  are  not  available,  or  if  contractor  facilities  provide  a  ready 
capability  not  available  in  government  facilities.  If  the  test  concept  calls  for  testing  at  contractor  or  non-MRTFB  facilities,  the 
SM  must  include  these  requirements  in  the  request  for  proposal  (RFP),  and  explain  in  the  TEMP  why  these  facilities  are 
required. 

2.23.2.  Waivers.  If  use  of  non-MRTFB  or  contractor  facilities  is  planned  for  any  program,  the  SM  must  justify  that  use  with 
a  cost-benefit  analysis,  and  request  a  waiver  from  AFMC/DO.  The  waiver  must  be  approved  and  documented  in  the  TEMP 
before  SMs  obligate  the  government  to  using  non-MRTFB  or  contractor  facilities.  Provide  an  inform-ation  copy  of  the  waiver 
to  HQ  USAF/TE. 

2.23.3.  Scheduling  Test  Resources.  SMs,  RTOs,  and  OTAs  will  use  the  Air  Force  Priority  and  Precedence  Rating  System 
described  in  AFI  16-301,  US  Air  Force  Priority  System  for  Resources  Management,  as  a  guide  for  scheduling  MRTFB 
resources. 

Section  E — DT&E  Objectives 

2.24.  DT&E  Focus  on  the  System.  SMs,  contractors,  and  RTOs,  as  appropriate,  will  plan  for,  structure,  and  conduct 
DT&E  programs  to  provide  clear  assessments  of  system  performance  appropriate  to  each  phase  and  milestone.  Address  the 
following  areas: 

2.24.1.  General  Objectives  of  DT&E. 

•  Identify  potential  operational  and  technological  capabilities  and  limitations  of  the  selected  alternative  system  or 
concept  identified  in  the  Analysis  of  Alternatives  (AoA). 

•  Support  the  early  identification  of  cost-performance  trade-offs  by  providing  analyses. 

•  Support  the  identification  and  description  of  design  technical  risks. 

•  Identify,  evaluate,  and  resolve  system  deficiencies  and  enhancements  as  early  as  possible  using  operationally  relevant 
T&E. 

•  Assess  progress  toward  meeting  critical  operational  issues,  mitigation  of  acquisition  technical  risk,  achievement  of 
manufacturing  process  requirements,  and  system  maturity. 

•  Assess  validity  of  assumptions  and  conclusions  from  the  AoA. 

•  Provide  data  and  analysis  in  support  of  the  decision  to  certify  the  system  ready  for  dedicated  operational  test  and 
evaluation. 

•  Assess  progress  toward  meeting  information  system  security  requirements,  to  include  minimum  requirements  in  DoD 
Directive  5200.28,  Security  Requirements  for  Automated  Information  Systems. 

2.24.2.  System  Performance. 

•  Evaluate  system  compliance  with  specifications  in  relation  to  operational  require-ments,  life  cycle  costs,  and 
schedules. 

•  Identify,  evaluate,  and  resolve  system  deficiencies  and  enhancements  according  to  TO  00-35D-54,  USAF  Deficiency 
Reporting  and  Investigation  System. 

•  Perform  cost-schedule-performance  trade-offs. 

•  Evaluate  system  capability  operate  in  a  nuclear,  biological  and  chemical  (NBC)  environment. 

•  Evaluate  system  vulnerability  or  lethality  during  LFT&E. 

•  Evaluate  system  performance  envelopes  and  limitations,  projected  effectiveness,  and  safe  operating  parameters  under 
operationally  relevant  conditions,  to  include  survivability. 

2.24.3.  System  Compatibility  and  Interoperability. 

•  Evaluate  system  compatibility  and  interoperability  with  existing  and  planned  equip-ment  or  systems  using  logistics  test 
and  evaluation  procedures,  and  the  integrated  logistics  support  (ILS)  elements  as  a  framework  for  evaluation. 

•  Evaluate  network  hardware  and  software  system  effectiveness,  suitability,  interoperability,  compatibility,  and 
integration. 

•  Track  and  evaluate  system  configuration  changes. 

2.24.4.  System  Suitability. 

•  Evaluate  system  reliability,  maintainability,  and  availability, 

•  Identify,  evaluate,  and  resolve  deficiencies  in  fielded  systems’  reliability  and  maintainability. 

•  Conduct  aging  and  surveillance  programs. 

•  Assess  system  supportability  and  identify  life  cycle  cost  drivers. 
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2.24.5.  System  Safety. 

•  Identify,  evaluate,  track,  and  resolve  all  safety  deficiencies  before  start  of  dedicated  OT&E. 

•  Ensure  the  system  is  safe  to  operate,  or  residual  hazards  have  been  mitigated  to  acceptable  levels  of  risk. 

2.24.6.  Soundness  of  System  Design. 

•  Ensure  engineering  and  systems  integration  is  complete. 

•  Identify  all  significant  design  problems,  necessary  changes,  and  retesting  to  verify  resolutions  to  deficiencies. 

•  Perform  validation  of  models  and  simulations  of  system  performance  with  actual  test  data. 

•  Determine  the  technical  risk  of  the  design  and  ways  to  mitigate  these  risks. 

•  Evaluate  the  system’s  integrated  diagnostics. 

•  Determine  human  engineering  aspects  of  the  system  and  limiting  factors. 

2.24.7.  System  Training  Requirements. 

•  Identify  and  develop  training  programs  and  technical  materials  to  support  the  system. 

•  Develop  and  procure  training  systems  (e.g.,  simulators  and  trainers)  to  emulate  the  characteristics  of  the  system. 

2.25.  Certification  of  System  Readiness  for  Dedicated  OT&E.  DoD  5000.2-R  requires  a  certifying  official  from  the 
developing  agency  to  formally  certify  systems  are  ready  to  enter  the  dedicated  phase  of  OT&E.  The  Program  Executive 
Oificer  (PEO),  Designated  Acquisition  Commander  (DAC),  or  equivalent  official  (the  “certifying  official”)  will  determine  the 
broad  scope  and  certification  requirements  for  certifying  system  readiness  to  begin  OT&E.  The  SM  is  responsible  for 
implementing  an  effective  system  certification  process  as  early  as  practical  in  new  development  programs.  Use  of  the 
certification  process  is  mandatory  for  reviewing  all  ACAT  programs.  See  AFMAN  63-119,  Certification  of  System  Readiness 
for  Dedicated  Operational  Test  and  Evaluation,  for  details. 


Chapter  3 

MANAGEMENT  RESPONSIBILITIES 

3.1.  Office  of  the  Under  Secretary  of  Defense  for  Acquisition  and  Technology,  Director,  Test,  Systems  Engineering 
and  Evaluation  (OSD/DTSE&E).  DTSE&E: 

•  Exercises  oversight  responsibility  for  major  defense  acquisition  programs,  or  any  program  in  which  the  Secretary  of 
Defense  or  Congress  has  special  interest  or  oversight. 

•  Formulates  overarching  DT&E  policy  and  procedures  for  DoD. 

•  Advises  the  Secretary  of  Defense  and  the  Under  Secretary  of  Defense  for  Acquisition  and  Technology  (USD(A&T)) 
on  DT&E  matters. 

•  Publishes  once  a  year,  with  the  Director,  Operational  Test  and  Evaluation  (DOT&E),  a  combined  list  of  OSD  T&E 
oversight  programs. 

•  In  conjunction  with  DOT&E,  approves  the  TEMP  for  OSD  T&E  oversight  programs. 

•  Participates  in  integrated  product  teams  (IPT)  as  required  for  program  planning  and  execution. 

3.2.  Director,  Operational  Test  and  Evaluation  (OSD/DOT&E),  DOT&E: 

•  Exercises  oversight  responsibility  for  major  defense  acquisition  programs,  or  any  program  in  which  the  Secretary  of 
Defense  or  Congress  has  special  interest  or  oversight. 

•  Prescribes  overarching  OT&E  and  live  fire  test  and  evaluation  (LFT&E)  policies  and  procedures  for  DoD. 

•  Monitors  and  reviews  all  LFT&E  and  OT&E  in  the  DoD. 

•  Prepares  an  annual  report  for  the  Secretary  of  Defense  and  Congress  summarizing  the  OT&E  activities  of  the  DoD 
during  the  fiscal  year. 

•  Publishes  once  a  year,  in  conjunction  with  the  DTSE&E,  a  combined  list  of  OSD  T&E  oversight  programs. 

•  In  conjunction  with  DTSE&E,  approves  the  TEMP  for  OSD  oversight  programs. 

•  Approves  OT&E  plans  for  oversight  programs. 

•  Participates  in  integrated  product  teams  (IPT)  as  required  for  program  planning  and  test  execution. 

3.3.  Chief  of  Staff  of  the  Air  Force  (CSAF).  CSAF: 

•  Ensures  resources  are  available  to  organize,  train,  and  equip  personnel  for  T&E  throughout  the  Air  Force. 

3.4.  Headquarters,  US  Air  Force  Director,  Test  and  Evaluation  (HQ  USAF/TE).  HQ  USAF/TE  will: 

•  Set  Air  Force  T&E  policy,  and  oversee  the  T&E  process  throughout  the  Air  Force. 
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•  Advocate  funding  for  T&E  resources,  and  ensure  the  T&E  infrastructure  is  properly  configured  for  future  systems 
testing. 

•  Review  all  Mission  Need  Statements  (MNS),  Program  Management  Directives  (PMD),  Analyses  of  Alternatives 
(AoA),  Operational  Requirements  Documents  (ORD),  and  other  T&E-related  documents  for  use  by  the  Air  Staff,  field 
agencies,  and  OSD. 

•  Approve  and  sign  TEMPs  before  Air  Force  Acquisition  Executive  (AFAE)  approval. 

•  Approve  all  requests  to  modify  the  requirements  for  DT&E. 

•  Participate  in  integrated  product  teams  (IPX)  as  required  for  program  planning  and  execution. 

•  Keep  the  CSAF  and  the  Secretary  of  the  Air  Force  (SAF)  informed  on  the  status  of  all  Air  Force  T&E. 

3.5.  Assistant  Secretary  of  the  Air  Force  for  Acquisition  (SAF/AQ).  SAF/AQ  will: 

•  Oversee  DT&E  on  all  acquisition  programs. 

•  Document  T&E  responsibilities  in  the  PMD. 

•  Assign  a  program  element  monitor  (PEM)  for  each  acquisition  or  sustainment  program. 

•  Assign  staff  to  participate  on  integrated  product  teams  (IPT)  as  required  for  program  planning  and  execution. 

3.6.  Headquarters,  Air  Force  Materiel  Command  (HQ  AFMC).  AFMC  will: 

•  Manage  the  Air  Force  DT&E  process. 

•  Assist  operating  commands  in  ORD  preparation  (see  paragraph  5.4). 

3.7.  Headquarters,  AFMC  Director  of  Operations  (HQ  AFMC/DO).  AFMC/DO  will: 

•  Serve  as  the  Air  Force  focal  point  for  the  conduct  of  all  Air  Force  DT&E  and  LFT&E. 

•  Establish  and  provide  for  DT&E  training,  organization,  and  resources. 

•  Designate  the  RTO  for  DT&E  if  the  SM  has  not  selected  an  RTO.  Approve  the  RTO  selected  by  the  SM,  if  appropriate. 

•  Assist  the  operating  commands  in  ORD  preparation,  and  provide  inputs  on  measures  of  effectiveness  (MOE)  to  include 
in  RCMs. 

•  Ensure  each  AFMC  center  establishes  a  center  test  authority  (CTA)  to  oversee  the  DT&E  process. 

•  Function  as  the  certifying  authority  for  T&E  Acquisition  Professional  Development  Program. 

3.8.  AFMC  Single  Manager  (SM).  With  respect  to  T&E,  the  SM  will: 

•  Provide  cost  and  technical  data  to  operating  commands  for  inclusion  in  the  Analysis  of  Alternatives  (AoA). 

•  Request  test  support  and  resources  sufficiently  early  to  allow  for  proper  planning. 

•  Contact  the  appropriate  SFTC  office  for  test  planning  consultation  for  all  new  programs  without  a  PMD-designated 
RTO  (and  prior  to  any  test  planning). 

•  Choose  the  most  appropriate  RTO  from  SFTC-supplied  list  of  candidates,  and  forward  the  candidate  RTO  to 
AFMC/DO  for  approval. 

•  Form  integrated  product  teams  (IPT)  as  required  for  program  planning  and  execution. 

•  Ensure  all  DT&E  is  conducted  according  to  approved  test  plans. 

•  Prepare  the  TEMP. 

•  Chair  the  Test  Planning  Working  Group  (TPWG). 

•  Make  acquisition  documentation  available  to  DT&E  testers,  SFTC  offices,  and  users. 

•  Use  the  ORD,  MNS,  and  threat  assessment  documents  to  translate  system  require-ments  into  performance-based 
specifications  and  standards  during  each  acquisition  phase. 

•  Ensure  the  appropriate  AFMC  test  center  or  RTO  is  cognizant  of  each  DT&E  project  or  acquisition  program. 

•  Develop  and  maintain  an  integrated  test  and  evaluation  program  (ITEP)  to  provide  overall  system  life  cycle  test 
planning  and  documentation  for  fielded  systems. 

•  Form  and  chair  IPTs  (such  as  the  Material  Improvement  Program  Review  Board  (MIPRB)  and  the  Joint  Reliability  and 
Maintainability  Evaluation  Team  (JRMET))  to  track  and  resolve  deficiencies. 

•  Ensure  knowledgeable  and  objective  testers  (preferably  SFTC  or  RTO  representatives)  are  members  of  the  RFP,  source 
selection,  and  integrated  product  teams. 

•  Ensure  government  access  to  contractor  T&E  data,  deficiency  reporting  processes,  and  test  results  through  the  contract 
proposal  process. 

•  Certify  systems  are  ready  for  dedicated  OT&E  according  to  AFMAN  63-119. 

3.9.  AFMC  Single-Face-to-Customer  (SFTC)  Offices.  The  SFTC  offices  will: 

•  Facilitate  initial  OT&E  planning  for  new  programs,  major  modifications,  upgrades,  or  product  improvement  programs. 
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•  Facilitate  OT&E  investment  planning  activities  through  the  Technical  Planning  Integrated  Planning  Teams  (TPIPT). 

•  Serve  as  test  consultants  able  to  identify  available  resources  and  risks  in  test  options  (see  AFI  99-109,  Test  Resource 
Planning). 

•  Help  manage  test  planning  and  develop  test  documents  prior  to  designation  of  RTO. 

•  Provide  a  prioritized  list  of  recommended  RTOs  to  the  SM  if  the  PMD  has  not  previously  designated  the  RTO. 

•  Transfer  DT&E  planning  responsibility  to  the  RTO,  once  assigned. 

•  Document  and  implement  Air  Force  T&E  processes. 

3.10.  AFMC  Air  Logistics  Centers  (ALC)  and  Product  Centers.  Each  center  commander  will: 

•  Designate  a  Test  Representative  as  the  single  focal  point  for  T&E. 

•  Establish  procedures  for  implementing  the  T&E  process  consistent  with  this  AFI. 

•  Develop  policy  and  procedures  for  conducting  thorough  technical  and  safety  reviews  for  resident  T&E  programs. 

•  Provide  T&E  support  to  SMs,  RTOs,  AFOTEC,  and  operating  commands  as  required  in  each  ITEP,  PMD,  and  TEMP. 

•  Ensure  all  necessary  DT&E  is  conducted  in  support  of  fielded  systems  during  their  life  cycles. 

•  Participate  in  integrated  product  teams  (IPT)  as  required  for  program  planning  and  execution. 

3.11.  Test  Centers.  The  Air  Force  Flight  Test  Center  (AFFTC),  Air  Force  Development  Test  Center  (AFDTC),  and  Arnold 
Engineering  Development  Center  (AEDC)  will: 

•  Conduct  DT&E  using  the  T&E  process  and  according  to  this  AFI. 

•  Conduct  long  range  planning  to  ensure  adequate  T&E  infrastructure  and  processes  are  in  place  to  support  future 
technologies  and  acquisition  programs. 

•  Establish  policy  and  procedures  for  conducting  thorough  technical  and  safety  reviews  for  all  T&E  conducted  at  the 
center. 

•  Participate  in  initial  T&E  planning  for  new  programs,  major  modifications,  upgrades,  and  product  improvement 
programs. 

•  Participate  in  all  T&E  resource  investment  planning  processes,  groups,  boards,  etc, 

•  Provide  support  to  the  SFTCs  and  SMs  on  existing  and  planned  test  capabilities, 

•  If  designated  the  RTO  or  PTO,  provide  T&E  support  to  SMs  and  AFOTEC  as  required  in  the  PMD,  TEMP,  and 
paragraph  3.12. 

•  Participate  in  test  planning  working  groups  (TPWG)  and  integrated  product  teams  (IPT)  as  required  for  test  planning 
and  execution. 

3.12.  Responsible  Test  Organization  (RTO).  The  RTO  will: 

•  Manage,  plan,  conduct,  and  report  on  DT&E  according  to  the  T&E  process  and  this  AFI. 

•  Formulate  an  integrated  test  approach  and  develop  a  test  concept. 

•  Provide  test  results  to  the  SM  and  other  required  organizations  for  DT&E  of  systems,  subsystems,  and  components. 

•  Inform  HQ  USAF/TE  within  24  hours  of  all  significant  test  events  for  OSD  oversight  programs. 

•  Integrate  government  and  contractor,  and  DT&E  and  OT&E  requirements  for  combined  test  programs. 

•  Designate  and  define  test  responsibilities  and  oversee  the  activities  of  participating  test  organizations  (PTO). 

•  Support  test  programs  for  laboratories,  program  offices,  or  other  organizations  requiring  DT&E. 

•  Oversee  DT&E  safety.  See  AFI  91-202,  The  US  Air  Force  Mishap  Prevention  Program,  and  AFI  31-702,  System 
Security  Engineering,  on  safety  engineering. 

•  Establish  procedures  for  analyzing  hazards  and  risks  at  each  level  of  testing  so  DT&E  is  conducted  at  the  lowest 
feasible  risk. 

•  Review  draft  ORDs  to  ensure  the  test  community  can  effectively  and  efficiently  evaluate  the  system  against  the  user’s 
requirements. 

•  As  members  of  the  RFP  and  source  selection  teams,  evaluate  the  technical  aspects  of  the  RFP  and  proposals  to  ensure 
they  contain  clear  test  requirements.  Ensure  these  documents  reflect  the  agreed  test  concept. 

•  Participate  in  Test  Planning  Working  Group  (TPWG)  and  other  IPT  meetings  as  required. 

•  Use  the  approved  ORD  and  system  specifications  to  develop  test  plans  covering  the  users’  requirements  and  priorities. 

3.13.  Participating  Test  Organization  (PTO).  The  PTO  will: 

•  Assist  the  RTO  in  accomplishing  DT&E  as  directed. 

•  Participate  in  integrated  product  teams  (IPT)  as  required  for  program  planning  and  execution. 


3.14.  Air  Force  Operational  Test  and  Evaluation  Center  (AFOTEC).  AFOTEC  will: 
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•  Initiate  early  participation  in  DT&E  planning  and  development  processes. 

•  Participate  as  early  as  possible  in  TPWGs  and  other  IPTs  in  order  to  clarify  system  requirements. 

•  Prepare  Section  IV  of  the  TEMP. 

•  Conduct  early  operational  assessments  (EGA)  as  required. 

•  Advise  the  SM  and  RTO  about  making  DT&E  more  operationally  relevant. 

•  Participate  in  the  certification  of  system  readiness  for  OT&E  according  to  AFMAN  63-119. 

•  Participate  in  combined  DT&E/OT&E  to  the  maximum  extent  practical  without  compromising  the  independence  of 
dedicated  OT&E. 

3.15.  Operating  Command.  The  operating  command  (also  called  the  user  or  using  command)  will: 

•  Provide  up-to-date  operational  requirements  documents  (ORD)  and  concepts  of  operation  (CONOPS)  to  the 
developing  command  and/or  RTO. 

•  Participate  as  early  as  possible  in  the  development  of  the  TEMP  and  DT&E  plan  where  necessary. 

•  Develop  an  Analysis  of  Alternatives  (AoA)  or  other  similar  study  to  assist  the  SM  when  making  trade-offs  between 
cost,  schedule,  and  performance. 

•  Make  command  resources  available  to  support  T&E  when  possible  without  adversely  impacting  their  mission. 

•  Participate  in  integrated  product  teams  (IPT)  as  required  for  program  planning  and  execution. 

3.16.  Test  Planning  Working  Group  (TPWG).  The  TPWG  will: 

•  Support  the  SM  in  developing  and  maintaining  the  TEMP. 

•  Participate  in  setting  test  objectives,  MOEs,  and  evaluation  baselines. 

•  Define  organizational  responsibilities  and  relationships  for  testing. 

•  Help  estimate  T&E  costs. 

•  Develop  a  realistic  T&E  schedule. 

•  Determine  required  test  resources. 

•  Review  and  help  prepare  the  test  portions  of  RFPs  and  related  contractual  documents. 

•  Help  evaluate  contractor  proposals. 

•  Help  the  SM  set  the  scope  and  objectives  of  DT&E. 

•  Coordinate  resource  availability,  including  necessary  facility  upgrades  and  personnel  requirements, 

•  Form  subgroups  such  as  a  Computer  Resources  Working  Group  (CRWG)  to  address  software  T&E  requirements. 

•  Participate  in  the  test  readiness  review  (TRR). 

•  Form  a  subgroup  (e.g.,  the  Logistics  Test  Working  Group  (LTWG))  to  address  logistics  T&E  requirements,  schedule, 
planning,  execution,  and  reporting.  This  group  will  also  communicate  to  the  TPWG  the  logistics  resources  needed  to 
support  ground  tests,  logistics  tests,  flight  tests,  and  logistics  T&E. 

3.17.  Air  Force  Communications  Agency  (AFCA).  With  respect  to  the  T&E  of  new  network  hardware  and  software 
components,  the  AFCA  will: 

•  Serve  as  the  Air  Force  focal  point  for  planning  and  conducting  QT&E  for  these  systems. 

•  Participate  as  early  as  possible  in  the  development  of  the  TEMP  and  DT&E  plan  where  necessary. 

•  Serve  as  Executive  Agent  for  the  Interoperability  Testing  Steering  Group. 

•  Develop  and/or  coordinate  compatibility,  interoperability,  and  integration  policies,  procedures,  programs  among  Air 
Force  organizations. 

•  Participate  in  integrated  product  teams  (IPT)  as  required  for  program  planning  and  execution. 

3.18.  Other  Commands  or  Agencies,  The  participating  and  supporting  commands,  and  the  OTA  participate  in  DT&E  and 
supporting  IPTs  as  specified  in  the  PMD,  the  TEMP,  and  this  instruction. 


Chapter  4 

DT&E  IN  THE  ACQUISITION  PROCESS 


Section  A — General  Information 


4.1.  DT&E  Support  of  the  Acquisition  Process.  The  acquisition  process  consists  of  four  program  milestones  (MS)  and 
four  event-driven  phases  (see  DoD  5000.2-R  for  details).  The  milestones  are  key  points  at  which  decision  makers  review  a 
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program  and  authorize  advancement  to  the  next  acquisition  phase.  Since  DT&E  results  greatly  influence  these  decisions, 
planning  of  the  DT&E  strategy  must  begin  as  early  as  possible  to  mitigate  program  risks.  Specifically,  SMs  will  structure 
DT&E  to  reduce  program  risk,  assess  attainment  of  technical  performance  parameters,  and  provide  essential  information  to 
decision  makers  in  support  of  the  acquisition  process. 

4.2.  Single  Manager  (SM)  Responsibilities.  SMs  will  identify  critical  system  characteristics,  objectives,  T&E 
methodologies,  and  evaluation  criteria  during  the  initial  phases  of  an  acquisition  program.  These  criteria  will  help  determine 
the  testing  required  for  each  phase  of  the  acquisition  process,  and  will  provide  the  structure  to  guide  the  test  program.  SMs 
will  report  T&E  results,  including  system  capabilities,  limitations,  and  their  resolution,  before  each  MS  to  ensure  the  program 
meets  its  objectives  and  exit  criteria  before  starting  the  next  phase. 

Section  B — DT&E  During  the  Four  Acquisition  Phases 

4.3.  Pre-Milestone  0.  The  Air  Force  uses  the  Mission  Area  Assessment  (MAA)  and  Mission  Need  Analysis  (MNA) 
processes  to  identify  mission  needs  and  develop  operational  requirements.  Using  the  “strategy-to-task*'  framework,  users 
prepare  mission  need  statements  (MNS)  if  materiel  solutions  are  warranted.  Milestone  (MS)  0,  Approval  to  Conduct  Concept 
Studies,  marks  the  initial  formal  interface  between  the  requirements  generation  and  the  acquisition  management  systems,  and 
sets  Phase  0  in  motion. 

4.4.  Phase  0  (Concept  Exploration). 

4.4.1.  Planning  the  Test  Approach.  The  operating  command  will  explore  materiel  alternatives.  DT&E  agencies  will: 

•  Use  laboratory  testing,  modeling,  and  simulations  to  assess  the  most  appropriate  concepts,  technologies,  and  designs 
based  on  the  deficiencies  identified  in  the  MNS,  and  the  concepts  proposed  in  the  MS  0  acquisition  decision 
memorandum  (ADM). 

•  Validate  test  requirements,  and  identify  and  plan  resources. 

•  Earmark  realistic  test  environments  for  subsequent  DT&E  (e.g.,  ranges,  targets,  threat  simulators,  logistics  test 
requirements)  to  ensure  needed  T&E  infrastructure  and  resources  are  funded,  acquired,  or  upgraded. 

•  Formulate  an  integrated  test  strategy  for  inclusion  in  the  acquisition  strategy. 

4.4.2.  Exiting  Phase  0.  Decision  makers  use  T&E  results  from  previous  development  efforts  to  support  MS  I,  Approval  to 
Begin  a  New  Acquisition  Program.  A  favorable  MS  I  decision  establishes  a  new  acquisition  program  and  authorizes  entry  into 
the  next  phase. 

4.5.  Phase  I  (Program  Definition  and  Risk  Reduction). 

4.5.1.  Program  Initiation.  During  this  phase  the  RTO  will: 

•  Conduct  DT&E  on  subsystems,  components,  prototypes,  or  preproduction  articles  to  gather  data  to  support  calculating 
system  performance,  effectiveness,  suitability,  and  supportability. 

•  Refine  or  validate,  as  appropriate,  computer  models  and  simulations  to  determine  a  preferred  technical  approach. 

•  Identify  design  risks  and  recommend  testing  to  the  SM  to  reduce  those  risks  to  acceptable  levels. 

•  Ensure  the  critical  system  characteristics  and  critical  technical  parameters  to  be  tested  from  the  approved  ORD  have 
been  formalized,  and  provide  detailed  information  on  the  overall  test  schedule  and  required  resources  for  Phase  II. 

•  Assess  the  system  for  potential  live  fire  test  vulnerabilities  and  susceptibilities  needing  correction  before  or  during 
Phase  11. 

•  Evaluate  the  system  against  its  technical  performance-based  requirements. 

4.5.2.  Early  Operational  Assessment  (EOA).  While  Phase  I  testing  is  primarily  DT&E,  OTAs  may  examine  the  military  utility 
of  the  system  and  the  proposed  technical  approach  using  an  EOA.  (See  AFI  99-102.) 

4.5.3.  Exiting  Phase  1.  The  SM  will  compare  the  DT&E  results  with  predetermined  exit  criteria  (see  DoD  5000.2-R),  and 
inform  the  decision  makers  responsible  for  MS  II,  Approval  to  Enter  Engineering  and  Manufacturing  Development. 

4.6.  Phase  II  (Engineering  and  Manufacturing  Development). 

4.6.1.  Approving  a  System  for  Production.  During  Phase  II,  DT&E  and  OT&E  will: 

•  Evaluate  the  system  against  contract  requirements  and  performance-based  operational  requirements. 

•  Employ  a  combined  DT&E/OT&E  approach  to  the  maximum  extent  practical  in  order  to  avoid  duplication  of  effort. 

•  Identify,  track,  and  resolve  deficiencies  using  an  approved  deficiency  reporting  system. 

•  Assess  manufacturing  processes  with  respect  to  system  tolerances,  performance,  and  quality. 

•  Demonstrate  designs  are  stable,  operationally  effective,  logistically  supportable,  and  capable  of  being  produced 
efficiently. 
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•  Arrange  for  interoperability  certification  of  C^\  systems. 

•  Determine  the  lethality  of  a  munition  system  or  the  survivability  of  a  combat  system  in  accordance  with  live  fire  test 
(LFT)  requirements. 

4.6.2.  Exiting  Phase  II.  The  SM,  RTO,  and  the  OTA  will  provide  DT&E  and  OT&E  results  to  support  MS  III,  Production  or 
Fielding/Deployment  Approval.  The  production  decision  can  be  either  low  rate  initial  production  (LRIP)  or  full  rate 
production.  Decision  makers  use  Phase  II  T&E  results  to  ensure: 

•  Updated  configuration  baselines  and  cost  information. 

•  Validated  system  threat  assessments. 

•  Proper  execution  of  operational  support  plans  to  include  transition  from  contractor  to  in-house  support,  if  appropriate. 

•  Identification  of  operational  and/or  support  problems. 

4.7.  Phase  III  (Production,  Fielding/Deployment,  and  Operational  Support). 

4.7.1.  Producing  and  Fielding  the  System.  During  production,  fielding,  and  deploy-ment,  DT&E  will  help  ensure  the  system’s 
continued  ability  to  perform  as  intended,  provide  feedback  to  update  and  validate  system  threat  assessments,  correct  identified 
deficiencies,  and  ensure  support  resources  are  acquired. 

4.7.2.  Sustaining  the  System.  The  SM  may  continue  to  modify  or  upgrade  systems  during  the  sustainment  portion  of  the 
system  life  cycle  (generally  after  MS  III)  to  extend  useful  military  life,  enhance  producibility,  or  expand  operational  envelopes 
and  capabilities.  DT&E  during  sustainment  is  planned,  conducted,  and  reported  following  the  same  test  process  guidance  as 
is  applicable  to  all  DT&E.  Systems  may  re-enter  the  acquisition  system  at  a  milestone  (e.g.,  MS  II)  commensurate  with  the 
acquisition  strategy  for  needed  follow-on  development  or  sustainment  activities.  During  this  period,  the  SM  will: 

•  Update  and  validate  models. 

•  Continually  identify  and  correct  quality,  performance,  reliability,  maintainability,  and  safety  deficiencies. 

•  Start  an  aging  or  surveillance  program. 

•  Evaluate  product  improvement  changes  and  engineering  modifications. 

•  Conduct  additional  DT&E  and  production  acceptance  testing  (PAT)  to  confirm  and  monitor  performance  and  quality, 
and  verify  the  correction  of  deficiencies. 

•  Undertake  modification  or  upgrade  efforts  when  appropriate. 

4.7.3.  Exiting  Phase  III.  The  SM  must  ensure  systems  are  properly  demilitarized  and  disposed  of  at  the  end  of  their  useful 
life. 


Chapter  5 

KEY  DOCUMENTS  AND  THEIR  RELEVANCE  TO  THE  DT&E  PROCESS 

5.1.  Responsibilities  for  Documentation.  The  SM  will  ensure  a  direct  link  and  consistency  between  program  acquisition 
documentation  and  DT&E  planning,  conduct,  and  reporting.  DT&E  actions  will  directly  support  the  verification  of  user 
requirements.  Early  DT&E  involvement  will  ensure  testers  have  a  clear  understanding  of  the  users’  needs  and  program 
decisions. 

5.2.  Mission  Need  Statement  (MNS).  MAJCOM  personnel  will  prepare  a  MNS  to  identify  and  document  mission 
deficiencies  requiring  materiel  and/or  software  solutions.  The  MNS  defines  projected  needs  in  generic  operational  terms 
without  referring  to  a  specific  system,  and  forms  the  basis  for  the  initial  identification  of  T&E  critical  issues.  Developmental 
testers  must  understand  the  needs  and  requirements  identified  in  the  MNS  before  developing  the  test  concept.  The  test  concept 
is  a  road  map  which  identifies  assumptions,  rationale,  test  structure,  timelines,  and  resources  to  provide  efficient  and  effective 
T&E  information  to  satisfy  user  needs. 

5.3.  Analysis  of  Alternatives  (AoA).  Formerly  called  the  cost  and  operational  effectiveness  analysis  (COEA),  the  AoA 
supports  MS  decision  reviews  by  showing  decision  makers  the  advantages,  disadvantages,  and  estimated  costs  of  promising 
alternatives  under  consideration.  This  analysis  is  mandatory  for  potential  ACAT I  programs  and  other  AC  AT  levels  as  directed 
by  the  service  acquisition  executive  (SAE).  The  operating  command  will  complete  the  AoA  before  MS  I,  and  update  it  for 
other  milestones  as  required  by  the  Milestone  Decision  Authority  (MDA).  The  AoA  provides  a  historical  record  of  the 
alternatives  considered  at  each  MS  decision  point.  The  measures  of  effectiveness  (MOE)  used  in  the  AoA  must  be  linked  to 
the  MOEs  used  in  the  ORD  and  other  program  documents.  T&E  programs  validate  the  assumptions  (such  as  weapon  accuracy, 
sortie  generation  rate)  used  in  AoA  tradeoffs.  The  AoA  must  be  reviewed  early  in  the  program  to  determine: 

•  If  the  MOEs  are  complete  and  testable,  and  support  the  alternative  chosen. 


AFI  99-101  1  November  1996 


19 


•  What  test  assets  are  needed. 

•  What  type  of  long-term  test  planning  is  needed. 

5.4.  Operational  Requirements  Document  (ORD).  The  ORD  amplifies  and  refines  the  MNS.  The  operating  command 
will  prepare  the  ORD  during  Phase  0,  Concept  Exploration.  Testable  MOEs  and  measures  of  performance  (MOP)  from  the 
AoA  will  be  included  in  the  ORD  and  TEMP.  The  ORD  is  solution  oriented  and  will  be  based  on  the  preferred  solution  selected 
during  AoA  accomplishment.  The  ORD  documents  how  users  will  operate,  deploy,  and  support  a  system,  and  provides  initial 
guidance  for  all  acquisition  agencies.  The  TEMP  and  the  system  contract’s  performance-based  requirements  will  be  written 
to  satisfy  ORD  requirements. 

5.5.  Requirements  Correlation  Matrix  (RCM).  The  RCM  is  an  attachment  to  the  ORD  and  provides  a  system  audit  trail 
of  the  capabilities  and  characteristics  identified  in  the  ORD.  It  provides  the  basis  for  user  needs  and  requirements  in  the  ORD 
to  be  included  in  the  Integrated  Program  Summary  (IPS),  TEMP,  and  Acquisition  Program  Baseline  (APB),  and  serves  as  the 
foundation  for  developing  the  System  Maturity  Matrix  (SMM)  (see  paragraph  5.13). 

5.6.  Test  and  Evaluation  Master  Plan  (TEMP).  The  TEMP  correlates  and  integrates  T&E  with  the  overall  acquisition 
program  strategy,  schedule,  and  other  program  documentation,  and  defines  the  critical  path  for  completing  test  and  evaluation. 
The  TEMP  will  place  the  most  emphasis  on  the  next  phase  of  system  development  rather  than  provide  a  historical  account 
of  program  progress.  The  TEMP  should  be  updated  prior  to  major  milestones,  program  baseline  changes,  and  when  there  have 
been  significant  changes  to  the  program.  The  SM  has  overall  responsibility  for  the  TEMP,  and  the  TPWG  helps  with  writing. 
Fielded  systems  may  have  several  active  TEMPs  at  the  same  time  to  support  multiple  modification  programs.  Key  elements 
of  the  TEMP  may  be  included  in  a  Single  Acquisition  Management  Plan  (SAMP)  if  appropriate  for  the  acquisition  program 
(see  paragraph  6.12,3).  The  TEMP  is  no  longer  required  once  a  program’s  development  is  completed  and  the  COIs  are 
satisfactorily  resolved.  (See  DoD  5000,2-R,  Appendix  III.) 

5.7.  Single  Acquisition  Management  Plan  (SAMP).  The  SAMP  is  a  comprehensive,  integrated  document  which  discusses 
all  relevant  aspects  of  a  program  in  support  of  a  MS  decision.  As  a  program  management  strategy  document,  it  consolidates 
required  documentation  and  is  tailored  for  the  specific  needs  of  the  program.  The  SAMP  is  structured  to  streamline  the 
oversight  and  statutory  requirements  contained  in  all  other  management  plans  for  all  levels  above  the  PEO  or  DAC. 

5.7.1.  Requirement  for  a  SAMP.  A  SAMP  is  required  by  SAF/AQ  for  all  ACAT I  and  II  programs,  and  is  optional  for  ACAT 
III  programs.  All  program  stakeholders  must  be  included  in  the  SAMP  IPT  process  to  assist  in  its  preparation.  Unique  program 
needs  and  sound  management  practices  may  dictate  additional  detailed  planning  below  the  level  appropriate  for  a  SAMP  (e.g., 
a  TEMP  may  still  be  required).  See  SAF/AQ’ s  Single  Acquisition  Management  Plan  Guide  for  further  guidance. 

5.7.2.  Incorporation  of  the  TEMP.  The  SM  and  program  IPT  may  incorporate  key  elements  of  the  TEMP  (Parts  II,  III,  IV,  and 
V)  into  a  SAMP  if  it  makes  good  sense  for  managing  the  program,  and  adequately  addresses  the  needs  of  the  T&E  community 
and  senior  decision  makers.  A  separate  TEMP  may  not  be  required  if  the  SAMP  adequately  captures  all  T&E  strategies  and 
requirements  necessary  for  program  execution.  However,  the  TEMP  will  remain  the  primary  T&E  management  document  if 
program  risks  are  high,  or  senior  decision  makers  require  the  document. 

5.8.  Integrated  Test  Plan  (ITP).  The  ITP  records  all  individual  DT&E  test  plans  for  the  contractors  and  the  government. 
It  is  a  detailed,  working-level  document  that  assists  in  identifying  risk  as  well  as  duplicative  or  missing  test  activities.  The  SM 
will  ensure  the  ITP  describes  the  total  test  management  effort  and  the  most  efficient  use  of  all  test  resources  at  the  subsystem 
and  system  levels. 

5.9.  Threat  Assessments.  DT&E  planners  will  consider  the  exploitable  features  and  threats  identified  in  the  System  Threat 
Assessment  Report  (STAR)  for  ACAT  1  programs,  or  the  System  Threat  Assessment  (STA)  for  ACAT  II  and  III  programs, 
when  defining  a  realistic  test  environment. 

5.10.  Acquisition  Decision  Memorandum  (ADM).  The  ADM  documents  the  decisions  made  and  exit  criteria  established 
at  a  milestone  decision  reviews  or  in-process  reviews.  It  specifies  what  must  be  done  in  the  next  acquisition  phase. 
Developmental  testers  will  be  cognizant  of  and  implement  the  decisions  documented  in  the  ADM. 

5.11.  Acquisition  Program  Baseline  (APB).  The  APB  documents  the  most  important  cost,  schedule,  and  performance 
parameters  identified  in  the  ORD/RCM,  and  performance  requirements  for  each  phase  of  the  program.  DT&E  personnel  must 
be  cognizant  of  the  information  documented  in  the  APB  because  it  acts  as  the  “contract”  between  the  SM  and  the  MDA. 
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5.12.  Program  Management  Directive  (PMD).  The  PMD  provides  official  Air  Force  direction  for  execution  of  approved, 
funded  programs  or  activities.  The  PMD  provides  only  top  level,  concise,  and  timely  program  specific  direction,  and  does  not 
authorize  expenditure  of  funds.  HQ  USAF  usually  prepares  the  initial  PMD  upon  MS  0  approval.  HQ  AFMC  will  provide  HQ 
USAF  with  pertinent  DT&E  information  for  the  PMD,  including  unique  areas  of  risk,  and  responsibilities  of  each  command 
or  agency  in  the  DT&E  program.  T&E  officials  are  required  to  coordinate  during  the  PMD  review  and  approval  cycle.  (See 
HOI  800-2,  Preparation  of  Program  Management  Directives.)  The  PMD  typically: 

•  Assigns  programmatic  responsibilities  to  implementing  organizations,  MAJCOMs,  field  agencies,  and  test 
organizations,  as  applicable,  to  integrate  their  activities. 

•  Identifies  review  requirements  and  approval  authorities. 

•  States  unique  program  objectives,  constraints,  and  thresholds. 

5.13.  System  Maturity  Matrix  (SMM).  The  SMM  links  user  requirements,  “allocated”  requirements,  and  system 
specifications  to  expected  test  results  to  be  achieved  over  time.  It  lists  critical  technical  and  operational  characteristics  of  a 
system  that  will  be  assessed  at  major  decision  or  event  milestones.  All  acquisition  programs,  with  the  exception  of  programs 
in  production  that  have  met  all  of  their  user  requirements,  require  an  SMM.  The  SMM  is  a  management  tool,  not  a 
requirements  document.  The  SM  develops  the  SMM  in  coordination  with  the  user  and  OTA,  and  it  is  approved  by  the  PEO 
or  DAC.  Acquisition  personnel  will  use  the  SMM  to: 

•  Predict  system  performance  against  analytically  derived  interim  requirements,  APB  exit  criteria,  or  other  event 
milestones. 

•  Track  and  report  actual  progress  toward  verification  of  mature  system  performance. 

5.14.  Integrated  Program  Summary  (IPS).  Decision  makers  use  the  IPS  with  its  annexes  as  the  basis  for  decisions  at 
major  milestones.  The  SM  will  develop  the  IPS  to: 

•  Summarize  a  program’s  structure,  status,  assessments  and  plans,  as  well  as  recommendations  by  the  SM,  PEO  or  DAC, 

•  Identify  program  risk  areas  and  plans  for  reducing  risks. 

•  Provide  a  basis  for  establishing  program  cost,  schedule,  and  performance  objectives  and  thresholds  in  the  APB. 

•  Provide  a  basis  for  establishing  exit  criteria  for  the  next  acquisition  phase. 

5.15.  System  Engineering  Management  Plan  (SEMP).  The  SEMP  is  a  concise,  top-level  management  plan  describing  the 
integration  of  all  system  design  and  engineering  activities.  The  SM  ensures  the  technical  parameters  in  the  SEMP  are  linked 
to  those  in  the  TEMP  or  SAMP.  The  SM  uses  the  SEMP  to  describe: 

•  Requirements  analysis. 

•  Functional  analysis  and  allocation, 

•  The  system  configuration  management  process  and  the  control  of  program  baselines. 

•  Systems  analysis  and  control. 

•  The  system  design  evolution  process. 

•  Transitioning  critical  technologies. 

5.16.  Request  for  Proposal  (RFP).  The  RFP  is  the  formal  document  which  commun-icates  the  government’s 
performance-based  requirements  to  industry  and  solicits  proposals  to  satisfy  user  requirements.  SMs  must  describe  new 
systems  in  performance-based  terms  and  let  the  bidders  propose  the  “how-to”  details.  In  writing  RFPs,  SMs  must  minimize 
the  use  of  government  MIL-specs  and  standards  which  contain  overly  descriptive  process  standards  that  impede  access  to 
commercial  operations,  or  which  add  no  value.  Military  requirements  must  be  written  in  a  way  that  encourages  alternative 
solutions  (e.g.,  COTS  or  NDI)  and  reduces  OSD  oversight.  The  RFP  must  maximize  the  overlap  between  DoD  needs  and 
commercial  capabilities. 

5.17.  System  Requirements  Document  (SRD).  The  SRD  is  part  of  the  RFP.  It  details  the  performance-based  requirements, 
critical  design  requirements,  and  manufacturing  requirements  for  an  item,  material,  process,  or  service.  The  responsible 
engineering  organization  prepares  the  SRD  based  on  the  MNS  and  ORD.  The  SRD  also  establishes  the  criteria  by  which  the 
government  determines  whether  or  not  the  system  meets  the  performance  and  design  requirements.  Contractors  use  the  SRD 
to  develop  detailed  specifications  for  the  system. 

5.18.  Contract  Data  Requirements  List  (CDRL).  The  CDRL  identifies  and  describes  the  content,  format,  delivery 
instructions,  and  approval  and  acceptance  criteria  for  all  deliverable  contract  data  requirements.  In  preparing  the  CDRL,  the 
SM  will: 

•  Ensure  government  testers  concur  with  any  test-related  deliverable  data. 
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•  Ensure  government  testers  are  members  of  the  Data  Requirements  Review  Board  (DRRB),  The  DRRB  reviews  and 
recommends  approval  of  data  requirements  for  each  test  program. 

•  Consult  with  the  OTA  to  document  needed  test  data  requirements. 

•  Request  and  review  contractor  test  plans  and  procedures  well  before  the  actual  tests  to  allow  time  for  corrections  and 
modifications, 

•  Approve  contractor  test  plans  prior  to  the  start  of  test. 

•  Ensure  testers  oversee  all  contractor  testing. 

•  Request  the  contractor  T&E  results  in  time  for  program  decisions. 

5.19.  Integrated  Logistics  Support  Plan  (ILSP).  The  ILSP  is  the  government’s  formal  planning  document  for  logistics 
support.  It  describes  the  integrated  logistics  support  (ILS)  program  and  includes  ILS  requirements,  tasks,  and  milestones  for 
the  life  cycle  of  the  system.  The  ILSP  and  ORD  are  the  primary  source  documents  for  ILS  inputs  to  the  TEMP  or  SAMP. 

5.20.  Program  Introduction  Document  (PID).  The  PID  is  the  initial  planning  document  submitted  to  DoD  major  range 
and  test  facility  bases  (MRTFB)  to  formally  request  test  support.  The  SM  and  RTO  develop  PIDs  jointly,  starting  as  soon  as 
general  program  requirements  and  schedules  are  set.  SMs  usually  complete  PIDs  using  the  Universal  Documentation  System 
Range  Commanders  Council  Document  501-79,  or  an  alternate  format  approved  by  the  test  agency  or  supporting  range. 

5.21.  Statement  of  Capability  (SOC).  The  RTO  responds  with  a  SOC  to  a  request  for  test  support.  The  test  requester  and 
RTO  set  a  SOC  due  date  depending  on  the  complexity  and  requirements  of  the  test.  The  RTO  uses  the  SOC  to: 

•  Accept  the  test  program  and  provide  the  specified  levels  of  support. 

•  Accept  prerequisites  or  constraints  for  support. 

•  Provide  a  preliminary  cost  estimate. 

5.22.  Other  Formal  Agreements. 

5.22.1.  Memorandum  or  Letter  of  Agreement  or  Understanding  (MOA,  LOA,  MOU).  In  some  instances  a  PI  may  not  be 
appropriate  for  the  services  or  resources  required  of  a  particular  agency.  In  those  cases,  the  concerned  agencies  write  an  MOA, 
LOA,  or  MOU  to  outline  respective  tasks  and  responsibilities.  API  25-201,  Support  Agreement  Procedures,  gives  guidance 
on  the  format  and  content  of  these  documents. 

5.22.2.  Host-Tenant  Support  Agreement  (HTSA).  RTOs  or  PTOs  prepare  HTSAs  to  delineate  base  support  requirements,  and 
the  responsibilities  of  the  host  base  and  the  tenant  (the  test  team).  These  agreements  are  used  when  testers  plan  to  use  a 
non-MRTFB  location  which  does  not  have  an  established  process  for  granting  test  support.  API  25-201  gives  guidance  on  the 
format  and  content  of  these  documents. 


Chapter  6 
DT&E  PLANNING 


Section  A — Beginning  Stage 

6.1.  Supporting  the  Acquisition  Strategy.  The  DT&E  strategy  must  fully  support  the  selected  acquisition  strategy  while 
addressing  the  users’  needs.  To  meet  demands  for  reduced  time  and  cost,  modified  acquisition  strategies  such  as  COTS,  NDI, 
evolutionary,  and  incremental  acquisition  are  gaining  increased  emphasis.  The  user’s  mission,  priorities,  operational 
requirements  should  also  be  defined  before  developing  a  DT&E  test  strategy. 

6.2.  Integrated  Product  and  Process  Development  (IPPD).  IPPD  is  a  management  technique  that  integrates  all 
acquisition  activities  starting  with  requirements  definition  through  production,  fielding,  deployment,  and  operational  support 
in  order  to  optimize  the  design,  manufacturing,  business,  and  supportability  processes.  PMs  and  developmental  testers  will 
apply  the  concept  of  IPPD  throughout  the  acquisition  process  to  the  maximum  extent  practicable.  SMs  will  maximize  the 
overall  performance  of  the  entire  acquisition  system,  not  just  the  performance  of  individual  functional  areas,  by  identifying 
problems  early  and  maintaining  a  cooperative  spirit  of  problem  resolution.  (See  DoDD  5000.1  and  DoD  5000.2-R) 

6.2.1.  Integrated  Product  Teams  (IPT).  At  the  core  of  IPPD  are  integrated  product  teams  (IPT).  The  IPT  capitalizes  on  the 
strengths  of  all  participants  working  as  a  team  for  the  overall  benefit  of  the  program.  A  number  of  IPTs  may  work 
simultaneously  on  various  aspects  of  the  acquisition  program.  For  example,  the  Test  Planning  Working  Group  (TPWG)  is  a 
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specialized  IPT  addressing  test  matters  and  charged  with  developing  the  TEMP.  The  IPTs  report  their  results  up  to  an 
overarching  IPT  (OIPT)  which  consists  of  senior  decision  makers. 

6.2.2.  Integrated  Test  Planning,  The  SM  will  begin  an  integrated  test  planning  effort  to  plan  for  system  T&E  as  early  as 
feasible  in  the  acquisition  process.  The  SM  will  form  IPTs  involving  all  acquisition  players  (contractors,  subcontractors, 
developmental  and  operational  testers,  maintainers,  logisticians,  and  users).  The  SM  will  maintain  a  continuous  integrated 
planning  effort  using  IPTs  to  implement  the  test  program  and  adjust  planned  activities  when  unexpected  circumstances  arise. 
This  integrated  test  planning  must  consider  combined  testing  whenever  practicable. 

6.2.3.  Integrated  Test  Program  (ITP).  The  SM  will  develop  and  follow  an  ITP  approach  which  integrates  all  testing  in  order 
to: 

•  Ensure  the  required  resources  are  available. 

•  Prevent  test  duplication  through  use  of  a  combined  DT&E/OT&E  approach. 

•  Properly  integrate  the  DT&E  program  with  the  overall  acquisition  strategy. 

6.3.  Early  DT&E  Strategy  Planning.  The  SM  must  start  early  when  developing  a  DT&E  strategy  to  support  the  selected 
acquisition  strategy.  Modified  acquisition  strategies  such  as  evolutionary,  incremental,  COTS,  and  NDI  are  increasingly  used 
to  exploit  rapidly  advancing  technologies.  The  SM  should  consider  DT&E  strategies  with  more  combined  DT&E/OT&E, 
earlier  tester  involvement,  and  more  system  contractor  testing.  DT&E  strategies  are  developed  using  the  systems  engineering 
process. 

6.3.1.  What  is  Systems  Engineering.  Systems  engineering  is  both  a  technical  process  and  a  management  process.  The  SM  will 
use  the  systems  engineering  process  to  control  the  total  system  development  effort,  and  to  achieve  an  optimum  balance  of  all 
system  elements.  Systems  engineering  is  an  iterative  process  which  translates  operational  needs  into  descriptions  of  system 
parameters  in  the  form  of  specialized  program  plans  and  documentation.  Typical  outputs  are  the  SEMP,  TEMP,  ILSP,  interface 
control  documents,  risk  analyses  and  assessments,  DT&E  test  plans,  trade  studies,  design  documentation,  software 
development  plans,  and  producibility  plans.  The  SM  integrates  these  documents  to  optimize  overall  system  effectiveness. 

6.3.2.  Role  of  T&E  in  Systems  Engineering.  The  SM  will  ensure  system  definition  and  design  reflect  the  technical 
requirements  for  all  system  elements  such  as  equipment,  software,  personnel,  facilities,  and  data.  Testers  will  ensure  these 
technical  requirements  are  quantifiable  and  testable  within  the  time  and  funding  allowed.  Testers  will  identify,  define,  and  set 
priorities  for  all  required  DT&E,  identify  required  test  resources,  then  design,  plan  for,  and  conduct  experiments  to  influence 
system  design  and  verify  performance  predictions. 

6.4.  Modeling  and  Simulation  (M&S)  During  DT&E.  Modeling  and  simulation  are  integral  parts  of  the  test  process  used 
to  supplement  and  complement  open-air,  hardware-in-the-loop,  installed  system  facility  testing,  measurement  facility  testing, 
and  integration  laboratory  testing.  The  SM  is  responsible  for  planning  and  executing  all  required  M&S  support  during  DT&E. 
The  RTO  will  support  the  SM’s  development  and  application  of  standard  digital  system  models  (DSM)  for  the  system  under 
test.  The  RTO  will  assist  in  model  verification,  validation,  and  accreditation  according  to  AFI  16-1001,  Verification, 
Validation,  and  Accreditation. 

6.5.  Protecting  Critical  Information.  The  SM  will  plan  and  provide  for  the  protection  of  classified  and  sensitive 
unclassified  information,  including  information  communicated  electronically.  Radiation  or  emissions  from  test  instrumenta¬ 
tion  must  be  protected  within  the  guidelines  of  the  security  classification  guide,  and  international  treaties  or  agreements.  The 
SM  and  the  TPWG  will  address  procedures,  resources,  and  operations  security  (OPS EC)  measures  in  the  TEMP.  SMs  will 
ensure  programs  involving  critical  technologies  or  war  fighting  capabilities  have  an  approved  program  protection  plan  before 
testing  begins.  This  plan  will  be  compared  to  a  test  facility’s  security  plan  before  beginning  any  test  at  a  facility. 

Section  B — Developing  the  Test  Concept 

6.6.  Available  Information.  Sources  for  information  include  the  SFTC  offices,  the  test  centers,  the  Air  Force  Acquisition 
Model,  the  users,  the  Acquisition  Logistics  Lessons  Learned  System,  and  the  OT&E  Lessons  Learned  Data  Base.  When 
defining  DT&E  requirements  and  outlining  the  DT&E  test  concept,  research  available  sources  for: 

•  The  program’s  overarching  acquisition  strategy  and  T&E  strategy. 

•  Changes  to  the  acquisition  strategy. 

•  Technical  information  from  the  Defense  Technical  Information  Center  (DTIC). 

•  Information  regarding  T&E  of  comparable  systems. 

•  Lessons  learned  by  previous  acquisition  personnel. 

6.6.1.  Single-Face-to-Customer  (SFTC)  Office  Roles.  The  SFTC  offices  will  facilitate  early  DT&E  planning  for  new 
programs,  major  modifications,  product  improvement  programs,  and  T&E  investment  planning.  The  SFTC’s  goal  is  to 
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improve  the  efficiency  and  effectiveness  of  Air  Force  T&E  by  assisting  customers  in  the  disciplined  application  of  the  test 
process  (detailed  in  the  Air  Force  T&E  Process  Manuals).  The  SFTCs  also  identify  risks  in  the  test  options  available  to  the 
customer,  and  help  customers  understand  the  capabilities,  limitations,  and  applications  of  available  resources.  Once  testers 
complete  early  DT&E  planning,  the  SFTC’s  role  diminishes  to  maintaining  cognizance  of  the  program  and  supporting  the 
RTO.  The  SFTC  and  RTO  will  work  closely  to  ensure  a  smooth  transition. 

6.6. 1.1.  SFTC  Offices.  Three  SFTC  offices  support  the  test  process: 

•  Aircraft-Propulsion/Avionics-Electronic  Warfare,  phone  (805)  275-9250  or  DSN  525-9250. 

•  ArmamentAVeapons-C^I,  phone  (904)  882-9650  or  DSN  872-3316. 

•  Space,  (505)  846-6080  or  DSN  246-6080. 

6.6. 1.2.  Designating  the  SFTC.  For  complex  weapon  systems  where  more  than  one  SFTC  is  involved,  the  SM  must  contact 
HQ  AFMC/DO  for  a  lead  SFTC  designation.  All  pre-MS  0  acquisition  organizations  will  work  with  the  appropriate  SFTC 
office  for  early  test  planning. 

6.6.2.  The  Air  Force  Acquisition  Model  (AFAM).  The  AFAM  is  a  personal  computer-based  system  managed  by  ASC/SYM 
(DSN  785-0423).  The  model  identifies  and  benchmarks  the  core  processes  of  acquisition  management,  and  communicates  best 
practices  and  lessons  learned.  It  describes  and  maps  out  the  entire  “cradle-to-grave”  acquisition  process,  incorporating  policy 
changes  not  yet  documented  in  Air  Force  Instructions.  The  AFAM  also  contains  the  Air  Force  Lessons  Learned  database  which 
contains  technical  and  management  lessons  learned  in  the  acquisition  community. 

6.6.3.  Automated  Test  Planning  System  (ATPS),  ATPS  is  a  personal  computer-based  expert  system  to  guide  users  through  the 
preparation  or  review  of  TEMPs,  It  provides  structured  and  systematic  methods  for  TEMP  preparation,  review,  and  program 
risk  assessment.  ATPS  improves  TEMP  consistency,  quality,  and  efficiency,  and  helps  reduce  the  learning  curve  for  less 
experienced  testers.  Copies  can  be  obtained  from  OSD/DTSE&E(T&E)  at  DSN  225-7246. 

6.6.4.  OT&E  Lessons  Learned  Program.  HQ  AFOTEC’s  Policy  and  Procedures  Team  (AFOTEC/XRX,  DSN  246-5242) 
manages  the  OT&E  Lessons  Learned  Program.  An  abstract  listing  of  all  OT&E  lessons  learned  is  available  upon  request. 

6.7.  The  DT&E  Test  Concept.  The  SM,  with  advice  from  the  SFTC  and  RTO,  deter-mines  the  most  effective  test  concept 
for  reducing  risk  during  DT&E.  Once  formulated,  the  SM  implements  the  test  concept  using  an  integrated  test  program  (ITP) 
approach. 

6.8.  Risk  Reduction.  A  major  objective  of  DT&E  is  the  reduction  of  acquisition  risk.  To  assess  and  manage  risk,  SMs, 
RTOs,  and  other  acquisition  managers  shall  use  a  variety  of  techniques,  including  technology  demonstrations,  prototyping,  and 
test  and  evaluation.  Risks  must  be  well  understood  and  risk  management  approaches  developed  before  decision  authorities 
authorize  a  program  to  proceed  into  the  next  phase  of  the  acquisition  process, 

6.9.  Risk  Management  Program.  The  SM  will  establish  a  risk  management  program  to  be  applied  throughout  the  design 
process.  A  structured  process  will  be  established  for  reviewing  and  approving  all  test  plans  prior  to  the  conduct  of  any  DT&E. 
The  test  plan  review  process  will  include  thorough  technical  and  safety  reviews  conducted  according  to  this  instruction  and 
AFI  91-202,  The  US  Air  Force  Mishap  Prevention  Program.  Also  see  DoD  5000.2-R. 

6.10.  Combined  Testing.  A  combined  DT/OT  approach  will  be  used  to  the  maximum  extent  possible.  The  combined 
approach  must  not  compromise  either  developmental  or  operational  test  objectives,  A  final  independent  phase  of  OT&E  will 
be  required  for  beyond  low-rate  initial  production  decisions.  The  combined  DT&E/OT&E  strategy  will  be  directed  in  the  PMD 
and  described  in  the  TEMR 

6.10.1.  Order  of  Combined  Testing.  The  SM  will  generally  conduct  DT&E  first  to  answer  critical  technical  or  engineering 
questions,  and  determine  or  verify  the  system’s  performance  envelope.  DT&E  information  may  come  from 
contractor-conducted  and/or  government-conducted  tests.  Operational  assessments  (OA)  and  operational  utility  evaluations 
(OUE)  may  be  used  to  provide  early  operational  user  feedback  to  developers,  and  as  precursors  to  lOT&E.  OT&E  personnel 
will  participate  in  DT&E  to  familiarize  themselves  with  the  system  and  to  review  test  data  as  early  as  possible.  As  DT&E 
progresses,  developmental  and  operational  test  events  and  data  requirements  will  be  combined  where  possible. 

6.10.2.  Contractor  Involvement  in  Government  Testing.  Congressional  statute  (Title  10  Section  2399)  and  DoD  5000.2-R 
place  strict  limits  on  the  participation  of  system  contractors  in  OT&E.  The  SM,  RTO,  and  OTA  must  ensure  any  system 
contractor-generated  data  used  for  the  combined  portions  of  DT&E/OT&E  is  protected  from  tampering,  improper  collection 
methods,  manipulation,  bias,  or  conflict  of  interest.  The  SM  must  structure  the  DT&E  such  that  raw  data  is  accurate,  objective, 
and  reliable,  and  make  it  available  to  independent  testers  for  independent  evaluation. 

6.10.3.  Responsibility  for  Combined  Test  Results.  The  SM,  RTO,  and  the  OTA  must  take  steps  to  protect  contractor  DT&E 
data  by  playing  active,  informed  roles  in  test  planning  and  execution.  The  greater  the  reliance  on  contractors  for  combined 
test  data,  the  greater  the  need  for  oversight  by  knowledgeable  government  officials.  The  number  of  government  officials  is 
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a  management  decision  based  on  a  number  of  factors  such  as  the  technical  complexity  of  the  project  or  its  components,  the 
technical  qualifications  of  government  oificials,  and  the  inspection  techniques  available.  The  RTO  is  ultimately  responsible 
for  achieving  DT&E  objectives,  and  AFOTEC  or  the  designated  OTA  is  responsible  for  achieving  OT&E  objectives,  and  for 
evaluating  and  reporting  results  independently. 

6.10.4.  Combined  Test  Forces  (CTF).  Formation  of  a  CTF  will  be  directed  in  program  documentation  or  in  detailed 
memorandums  of  agreement.  All  test  organizations  participating  in  the  CTF  will  make  test  resources  available  under  a  unified 
command  structure  at  a  single  location,  or  as  agreed. 

6.11.  Contractor  and  Subcontractor  DT&E.  DT&E  contractual  requirements  must  be  consistent  with  established  HQ 
USAF/TE  and  AFMC  policies,  and  RTO  procedures.  The  RTO  will  review  all  contractor  test  plans  and  recommend  approval 
or  disapproval  to  the  SM.  The  SM  must  approve  those  plans,  procedures,  and  reports  (including  the  deficiency  reporting 
process)  verifying  the  system  or  subsystem  specifications.  The  SM  will  also  review  all  contractor  and  subcontractor  test  plans 
and  deficiency  reporting  processes  before  approval. 

6.12.  Test  and  Evaluation  Master  Plan  (TEMP).  The  SM  and  TPWG  will  document  in  the  TEMP  the  overarching 
strategy  for  test  planning,  funding,  and  execution.  The  TEMP  is  subject  to  HQ  USAF/TE,  OTA,  SAF/AQ,  and  OSD  review 
and  approval.  If  appropriate,  a  SAMP  may  be  used  in  lieu  of  the  TEMP  (see  paragraphs  5.6  and  5.7).  DT&E  will  not  start  until 
a  TEMP  (or  SAMP,  if  appropriate)  is  approved.  Joint  programs  require  multiple  Service  approval.  See  DoD  5000.2-R, 
Appendix  III,  for  details.  The  TPWG  will  use  the  TEMP  to: 

•  Fully  integrate  the  overall  program  acquisition  strategy,  test  concepts,  schedules,  funding,  and  associated 
documentation. 

•  Define  the  critical  path  to  achieving  each  acquisition  milestone. 

•  Accurately  reflect  the  user’s  requirements  and  MOEs,  and  how  they  will  be  tested. 

•  Provide  linkage  between  operational  requirements,  system  characteristics,  critical  parameters  (performance,  cost,  and 
schedule),  MOEs  used  in  testing.  Show  how  these  support  the  mission  or  task  the  system  is  to  perform. 

•  Document  how  testable  MOEs  and  MOPs  in  the  AoA  will  be  addressed  during  T&E. 

•  Document  RTO,  PTO,  and  OTA  responsibilities  and  key  organizational  relationships. 

•  Emphasize  the  next  phase  of  system  acquisition  rather  than  provide  an  historical  account  of  the  program. 

•  Describe  exit  criteria  documented  in  the  Acquisition  Decision  Memorandum  (ADM)  and  the  Acquisition  Program 
Baseline  (APB). 

•  Document  when  and  how  a  combined  test  approach  will  be  used  if  time  and  cost  savings  warrant. 

•  Document  the  impacts  of  test  and  resource  limitations. 

6.12.1.  TEMP  Preparation.  The  SM  will  develop  and  update  the  TEMP  in  support  of  program  milestones,  program  reviews, 
and  other  decision  points.  The  SM  will  use  an  integrated  product  team  (IPT)  approach  (e.g.,  a  TPWG)  when  developing  and 
updating  the  TEMP.  ATPS  is  available  for  creating  and  reviewing  TEMPs. 

6.12.2.  TEMP  Content.  The  TWPG  will  make  the  initial  TEMP  as  specific  as  possible.  As  the  system  progresses  through  its 
milestones  and  becomes  more  clearly  defined,  the  TEMP  should  become  more  detailed.  However,  the  TPWG  must  not  allow 
any  areas  to  remain  “to  be  determined”  (TBD)  in  the  DT&E  portion  of  a  TEMP  past  MS  L  See  DoD  5000.2-R,  Appendix  III, 
for  details  about  content  and  format.  The  completed  TEMP  will  not  exceed  30  pages  and  represents  agreement  among  all 
TPWG  participants. 

6.12.3.  Incorporation  of  the  TEMP  into  a  SAMP.  Key  elements  of  the  TEMP  may  be  incorporated  into  a  SAMP  at  the 
discretion  of  the  PM,  and  with  the  concurrence  of  the  TPWG.  This  consolidation  will  only  be  done  if  consistent  with  common 
sense  and  sound  business  practices.  If  appropriate,  the  TEMP  may  be  developed  as  an  annex  to  the  SAMP.  As  a  minimum, 
the  SAMP  (or  TEMP  annex)  must  capture  critical  T&E  information  needed  for  program  execution.  All  TPWG  stakeholders 
will  be  included  in  the  SAMP  IPT  when  preparing  the  T&E  portion  of  the  SAMP.  More  details  are  contained  in  the  Single 
Acquisition  Management  Plan  Guide,  29  April  1996,  available  from  SAF/AQXA. 

6.12.4.  TEMP  Submittal  and  Coordination.  The  SM  will  submit  the  TEMP  “in  parallel”  to  all  organizations  represented  on 
the  TPWG.  Organizations  are  normally  allowed  30  days  for  coordination,  and  TEMP  due  dates  are  firm.  Organizations  will 
not  withhold  TEMP  coordination  or  approval  in  an  attempt  to  resolve  issues  or  force  solutions.  If  issues  remain  unresolved 
for  more  than  30  days,  the  dissenting  organization  must  provide  the  SM  with  a  position  statement  or  a  formal  nonconcurrence. 
When  the  TEMP  cannot  be  submitted  on  time,  the  SM  must  write  to  the  Air  Force  approval  authority  stating  the  reason  for 
the  delay  and  commit  to  a  new  submittal  date.  TEMP  approval  authorities  will  not  approve  TEMPs  without  a  current  MNS, 
STAR  or  STA,  and  ORD. 

6.12.4.1.  Initial  TEMP  Submission.  SMs  will  submit  initial  TEMPs  via  the  program  element  monitor  (PEM)  to  the  PEG  or 
DAC,  and  the  Mission  Area  Director  (MAD).  The  MAD  will  coordinate  the  TEMP  through  the  Director,  Air  Force  Test  and 
Evaluation  (HQ  USAF/TE),  and  submit  it  to  the  Service  Acquisition  Executive  (SAF/AQ)  for  Service  approval. 
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6.12.4.2.  Submission  to  OSD.  TEMPs  requiring  OSD  approval  must  be  submitted  to  the  PEM  45  days  prior  to  the  OSD  due 
date  to  allow  a  minimum  of  30  days  for  internal  Air  Force  review,  incorporation  of  comments,  and  approval.  After  Service 
approval,  the  draft  TEMP  will  be  submitted  to  OSD  for  approval  45  days  prior  to  the  MS  review;  and  the  final  TEMP  will 
be  submitted  10  days  prior  to  the  MS  review.  Submission  to  two  OSD  offices  is  required:  the  Director,  Test,  Systems 
Engineering  and  Evaluation  (OSD/DTSE&E);  and  the  Director,  Operational  Test  and  Evaluation  (OSD/DOT&E). 

6.12.5.  TEMP  Updates  and  Changes.  After  initial  submission,  the  SM  revises  the  TEMP  to  support  milestones,  or  to  reflect 
significant  changes  or  breaches  in  the  program.  SMs  will  review  their  TEMPs  annually  as  part  of  the  PMD  cycle.  They  will 
submit  updates  or  changes  through  their  PEOs,  DACs,  and  MADs,  as  appropriate. 

6.12.6.  Test  Planning  Working  Group  (TPWG)  Functions.  The  TPWG  is  mandatory  for  major  programs  and  OSD  “oversight” 
programs.  The  SM  forms  the  TPWG  to  include  a  chairperson  from  the  program  office  and  representatives  from  each 
organization  involved  in  the  test  program.  TPWG  representatives  must  work  together  as  a  cross-functional  team  for  the  good 
of  the  program.  They  must  be  empowered  to  make  decisions  for  their  organizations,  be  highly  knowledgeable  in  their 
functional  areas,  and  provide  full  support  in  the  development  and  coordination  of  the  TEMP.  As  a  minimum,  the  following 
organiza-tions  will  be  represented: 

•  Program  office  functional  area  experts  (e.g.,  contracting,  logistics,  and  T&E. 

•  Single  Face  to  Customer  Office. 

•  Test  agencies  (including  RTO,  PTO,  and  OTA). 

•  The  Defense  Contract  Management  Command  (DCMC). 

•  Operating  command(s). 

•  Supporting  command(s). 

•  Participating  command(s). 

•  The  PEM. 

•  OSD/DTSE&E  and  OSD/DOT&E. 

•  HQ  USAF/TEP. 

•  System  contractor(s). 

•  Support  contractor(s). 

•  Additional  representatives  (when  appropriate): 

•  Subcontractor(s). 

•  Federal  Aviation  Administration  (FAA). 

•  National  Aeronautics  and  Space  Administration  (NASA). 

•  Any  other  organizations  involved  in  the  T&E  effort. 

6.13.  Developing  the  Request  for  Proposal  (RFP). 

6.13.1.  RFP  Special  Contract  Clauses.  Government  testers  will  review  the  RFP,  particularly  Section  H,  to  ensure  it  contains 
any  special  clauses  necessary  for  executing  the  approved  test  concept.  Government  testers  must  also  review  Section  L  to 
ensure  test  program  requirements  are  properly  addressed.  If  the  test  concept  requires  a  combined  DT&E/OT&E  approach, 
provisions  must  be  included  for  protecting  the  quality  and  integrity  of  contractor  test  data  for  later  use  during  independent 
OT&E  (see  paragraphs  6.10  and  2.2.4). 

6.13.2.  The  System  Requirements  Document  (SRD).  The  SM  will  involve  develop-mental  testers  in  identifying  the  T&E 
required  in  Section  4  of  the  SRD,  and  in  developing  the  verification  cross  reference  matrix.  Section  3  of  the  system 
specification  lists  all  the  system  requirements  testers  must  verify,  and  Section  4  lists  the  methods  testers  will  use.  The  RTO, 
along  with  the  OTA  and  other  test  organizations,  must  ensure  the  user’s  requirements  are  correctly  translated  into  meaningful 
and  testable  contractual  language. 

Section  C — Scheduling  Time  and  Resources 

6.14.  Formulating  a  Test  Program  Schedule.  The  SM  will  develop  the  test  program  schedule  for  inclusion  in  the  RFP 
and  the  TEMP.  During  the  initial  planning  stages,  the  SM  should  develop  a  top-level,  event-driven  master  schedule. 
Success-oriented  or  calendar-driven  schedules  must  be  avoided.  The  SM  must  get  estimates  from  the  contractor  on  when 
software,  models,  prototypes,  mockups,  etc.,  will  be  delivered,  and  get  estimates  from  the  RTO  and  OTA  on  the  DT&E  and 
OT&E  schedules,  respectively.  With  these  estimates,  the  SM  will  show  all  test  support  requirements.  To  reduce  risk,  adequate 
time  must  be  given  to  DT&E,  OT&E,  unplanned  contingencies,  review  of  DT&E  results,  and  correction  of  deficiencies 
discovered  during  DT&E.  As  the  planning  continues  and  the  test  program  matures,  the  master  schedule  will  be  expanded  into 
detailed  schedules. 


26 


AFI  99-101  1  November  1996 


6.15.  Resource  Planning.  The  SM  will  plan  all  needed  financial  resources  to  complete  DT&E  objectives  according  to  AFI 
65-601,  Volume  1,  Budget  Guidance  and  Procedures. 

6.15.1.  Test  Facilities.  The  SM  will  plan  to  use  Air  Force  facilities  first,  and  then  DoD  facilities,  to  the  maximum  extent 
possible.  See  paragraph  2.23  for  more  details. 

6.15.2.  Budgeting  for  Resources.  The  SM  will  budget  for  DT&E,  lOT&E  test  articles,  and  Type  1  training  as  required.  The 
OTA  will  budget  for  all  other  OT&E  resources,  including  training  not  specifically  budgeted  by  the  SM.  The  SM  will  add 
program-specific  test  resources  not  available,  funded,  or  programmed  to  the  program  baseline  and  includes  them  in  the 
resources  section  of  the  TEMP. 

6.15.3.  Using  Foreign  Materiel.  If  foreign  materiel  is  required  as  part  of  the  DT&E,  the  SM  and  RTO  will  comply  with  the 
foreign  materiel  requirements  outlined  in  AFI  99-114,  Foreign  Material  Program.  AFI  99-109,  Test  Resource  Planning, 
provides  further  guidance. 

6.15.4.  Test  Data  Resources.  The  RTO  will  identify  the  various  parameters  for  T&E,  and  specify  what  resources  are  needed 
to  measure,  record,  process,  and  track  these  parameters.  Telemetry  and  encryption  requirements  will  be  considered.  The  SM 
will  evaluate  the  resource  requirements  to  determine  the  optimum  balance  between  cost  and  schedule. 

6.15.5.  System  Security  Test  Resources,  SMs  and  RTOs  must  identify  system  security  resources  and  include  them  in  test 
planning  documents.  See  AFI  31-101,  Volume  1,  The  Physical  Security  Program. 

Section  D — Test  Support 

6.16.  Request  for  Test  Support.  After  consulting  with  the  SFTC,  the  SM  will  typically  nominate  one  of  the  following 
government  agencies  (among  others)  as  the  responsible  test  organization  (RTO): 

•  An  Air  Force  Test  Center  (AEDC,  AFDTC,  AFFTC). 

•  An  Air  Logistics  Center  (ALC)  Flight  Test  Squadron. 

•  An  Air  Force  Product  Center  T&E  directorate. 

6.16.1.  RTO  Designation.  The  RTO  is  the  lead  government  entity  that  is  qualified  and  responsible  for  DT&E.  If  the  PMD  does 
not  designate  an  RTO  for  DT&E,  the  SM  will  contact  the  appropriate  SFTC  office  for  advice.  The  SFTC  office  will  forward 
a  prioritized  list  of  recommended  RTOs  to  the  SM  and  AFMC/DO,  from  which  the  SM  will  nominate  an  RTO  and  forward 
it  to  HQ  AFMC/DO  for  approval.  HQ  AFMC/DO  will  then  designate  the  RTO  and  notify  SAF/AQ  for  documentation  in  the 
PMD,  During  the  development  of  a  system,  a  program  may  need  several  RTOs,  but  only  one  will  be  the  lead  RTO  for  a  specific 
test  program  element  at  any  given  time.  Although  contractors  are  not  eligible  to  be  RTOs,  some  RTO  tasks  may  be  delegated 
to  them  in  the  SOW. 

6.16.2.  Alternative  Designation  Procedures.  If  there  are  compelling  reasons  for  not  selecting  one  of  the  RTOs  from  the 
SFTC’s  list,  the  SM  will  forward  justification  for  an  alternate  to  AFMC/DO.  HQ  AFMC/DO  will  be  the  decision  authority 
and  officially  designate  the  RTO.  The  T&E  Mission  Element  Board  will  be  the  final  arbiter  for  any  further  disagreements  about 
RTO  selection. 

6.17.  Personnel  Training.  The  SM,  RTO,  and  OTA  will  articulate  any  unique  or  special  training  requirements  needed  for 
test  personnel  early  in  the  planning  process.  Personnel  must  complete  training  in  time  to  perform  testing.  The  SM  will  include 
appropriate  training  requirements  in  the  statement  of  work  (SOW)  portion  of  the  RFP. 

6.18.  DT&E  Plans.  The  number  and  complexity  of  DT&E  plans  needed  for  an  acquisition  effort  is  determined  by  the 
nature  of  the  system  and  the  types  of  testing  required.  Most  DT&E  plans  are  either  management  plans  or  detailed  test  plans, 
all  of  which  are  outlined  in  the  ITP. 

6.18.1.  Detailed  Test  Plans  (DTP).  DTPs  describe  specific  tests  and  how  they  will  be  conducted.  Contractors,  subcontractors, 
the  SPO,  the  RTO,  the  PTO,  the  OTA,  and  other  test  agencies  will  develop  DTPs  for  all  testing  regimens.  Contractors  should 
design  DTPs  to  satisfy  the  verification  requirements  of  the  SOW  and  RFP.  The  SM  will  develop  DTPs  to  demonstrate  or  verify 
contractor  compliance  with  contract  specifications  and  performance-based  requirements.  Operational  testers  will  develop 
DTPs  to  verify  operational  effectiveness  and  suitability  requirements  stated  in  the  ORD.  AFPAM  99-116,  Test  and  Evaluation 
Management  Guide,  provides  an  outline  for  preparing,  coordinating,  and  revising  a  DTP. 

6.18.2.  Integrated  Logistics  Support  (ILS)  Planning.  The  SM  ensures  the  test  program  evaluates  the  logistical  aspects  of 
systems  using  the  ILSP.  The  SM  also  integrates  testing  of  the  ILS  elements  into  the  overall  schedule.  The  ILS  manager  will 
participate  in  DT&E  and  OT&E  planning  by  developing  detailed  ILS  test  objectives  and  MOEs  for  each  acquisition  or 
modification  phase. 

6.19.  Technical  and  Safety  Reviews.  Government  test  plans  and  test  support  missions  are  subject  to  separate  technical  and 
safety  reviews  by  independent  government  test  personnel.  Reviews  will  be  conducted  as  necessary  for  commercial  or 
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non-govemment  systems  tested  in  DoD  T&E  facilities  or  when  DoD  assets  are  used.  Test  organizations  must  establish  and 
publish  procedures  for  determining  when  and  how  these  reviews  will  be  accomplished.  The  review  process  could  be  as  simple 
as  a  series  of  individual  reviews  for  simple  tests,  to  a  formal  Technical  Review  Board  (TRB)  and/or  Safety  Review  Board 
(SRB)  for  complex  tests  with  elevated  risk  levels. 

6.19.1.  Technical  Review  Board  (TRB),  The  TRB  assesses  the  soundness  of  system  designs  and  test  plans.  Technically 
qualified  personnel  with  test  management  experience,  but  who  are  independent  of  the  test  program,  will  perform  these 
reviews. 

As  a  minimum,  technical  reviews  will: 

•  Evaluate  test  requirements,  techniques,  approaches,  and  objectives. 

•  Assess  and  reduce  test  risk. 

6.19.2.  Safety  Review  Board  (SRB).  The  SRB  assesses  whether  the  test  plan  has  identified  and  mitigated  all  safety  hazards, 
and  whether  adequate  plans  exist  to  eliminate  or  mitigate  risks.  At  the  recommendation  of  the  SRB,  the  SM  and  RTO  will 
eliminate  or  mitigate  hazards  caused  by  the  operation  of  the  system,  system  design  deficiencies,  and  potential  unsafe  acts  or 
procedural  errors.  RTOs  will  set  up  procedures  for  controlling  and  supervising  tests  consistent  with  the  risk  involved  and 
according  to  local  range  safety  criteria.  See  AFI  91-202,  The  US  Air  Force  Mishap  Prevention  Program. 

6.19.3.  Nonnuclear  Munitions  Safety  Board  (NNMSB).  The  NNMSB  will  review  and  approve  all  live  munition  expenditures 
and  test  events  (including  explosive  devices  such  as  initiators  and  fuses).  AFI  91-205  discusses  the  NNMSB. 

6.19.4.  Determination  of  Risk.  During  safety  reviews,  safety  personnel  will  establish  safety  risk  levels  according  to 
MIL-STD-882.  Testing  organizations,  including  R&D  laboratories,  will  implement  a  safety  review  process  according  to  this 
standard.  The  safety  process  will  include  how  to  route  all  PI  documents  through  the  safety  office  when  developing  the  SOC. 

6.20.  Certification  of  System  Readiness  for  Dedicated  OT&E.  The  certification  process  provides  a  structured 
mechanism  for  identifying  and  reducing  risks  associated  with  transitioning  from  DT&E  to  OT&E.  The  SM,  with  the 
concurrence  of  the  “certifying  official,”  will  tailor  the  certification  process  to  meet  the  needs  of  the  acquisition  program.  The 
SM,  RTO,  OTA,  PTO,  operating  command(s),  and  other  supporting  commands  will  participate  in  the  process  according  to 
AFMAN  63-119. 

6.20.1.  Starting  Early.  SMs  will  implement  the  certification  process  as  early  as  practical  for  all  AC  AT  programs  which  plan 
for  dedicated  OT&E.  The  certification  process  is  a  continuous  effort,  not  a  single  event  in  time,  and  is  not  tied  to  any  particular 
acquisition  milestone.  However,  final  certification  for  dedicated  OT&E  must  be  done  a  minimum  of  30  days  prior  to  the 
planned  start  of  dedicated  OT&E. 

6.20.2.  Certifying  Officials.  The  PEO,  DAC,  or  equivalent  official  is  the  “certifying  official”  for  assigned  programs.  In  some 
cases,  the  SM  may  be  the  certifying  official.  The  certifying  official  will  determine  the  broad  scope  and  requirements  for 
certifying  system  readiness  to  begin  the  dedicated  phase  of  OT&E. 

6.21.  Software  Testing.  During  DT&E,  testers  will  set  quantitative  and  demonstrable  performance  objectives  for  software, 
and  structure  the  testing  to  demonstrate  software  has  reached  a  level  of  maturity  appropriate  for  each  phase. 

6.22.  T&E  Support  Aircraft.  On  the  advice  of  the  RTO,  field  commanders  will  review  and  approve  the  use  of  chase 
aircraft  to  support  Air  Force  and  contractor  testing  (see  AFI  16-402,  Aerospace  Vehicle  Assignment,  Distribution,  Accounting, 
and  Termination), 


Chapter  7 

CONDUCTING  AND  ANALYZING  TESTS 


Section  A — Getting  Ready  To  Test 

7.1.  The  Integrated  Test  Plan,  RTOs  will  execute  DT&E  according  to  the  approved  integrated  test  plan  (ITP)  and  detailed 
test  plans  (DTP).  The  SM  is  responsible  for  the  overall  test  program  and  must  keep  abreast  of  all  contractor,  RTO,  and  PTO 
test  efforts.  The  RTO  will  designate  a  DT&E  test  director  for  all  government  tests  who  will  control  the  DT&E  test  team  and 
associated  resources. 

7.2.  Test  Readiness  Reviews  (TRR).  The  SM  will  conduct  TRRs  before  executing  a  test  event  to  ensure  test  procedures 
comply  with  test  plans,  and  test  requirements  will  be  accomplished.  The  TRR’s  goal  is  to  satisfy  senior  commanders  that: 

•  The  system  is  ready  for  testing. 
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•  Pretest  analysis  has  predicted  test  results. 

•  Test  personnel  are  completely  prepared  for  the  test. 

•  Known  anomalies  have  not  increased  the  risks  associated  with  executing  the  test. 

•  All  reasonable  efforts  have  been  made  to  minimize  risk. 

•  Test  results  will  provide  the  appropriate  inputs  for  evaluation  of  the  tested  system. 

7.3.  Preparing  the  Test  Team,  The  SM  or  DT&E  test  director  will  fully  brief  all  DT&E  test  team  members  and  support 
personnel  on  the  test  program,  and  will  ensure  training  is  sufficient  to  conduct  the  test.  This  preparation  will  include 
discussions  of: 

•  The  test  program  and  its  objectives. 

•  The  test  items. 

•  Methods  for  conducting  the  tests. 

•  Program  schedules  and  major  milestones. 

•  Team  members*  responsibilities. 

•  Mishap  reporting. 

•  Safety  for  personnel  and  systems. 

•  Environmental  protection. 

•  Data  requirements. 

•  Data  collection  and  analysis  procedures. 

•  The  deficiency  reporting  system  (as  outlined  in  TO  00-35D-54). 

•  Responsibilities  for  preparing  the  DT&E  report. 

Section  B — Executing  Tests  and  Evaluations 

lA.  RTO  Execution  of  Test  and  Evaluation.  The  RTO  will  use  the  scientific  method  and  the  test  process  to  execute  tests 
by: 

•  Conducting  pretest  analysis  and  risk  analysis  to  refine  test  scenarios,  identifying  critical  test  points,  and  predicting 
system  performance. 

•  Conducting  test  sequences  under  progressively  more  strenuous  test  conditions. 

•  Processing  test  data  and  analyzing  the  results. 

•  Matching  test  results  to  pretest  predictions  and  modifying  models  prior  to  future  tests. 

•  Modifying  test  articles  and  prototypes  as  needed  to  fix  identified  deficiencies. 

•  Evaluating  test  results  to  determine  system  operational  effectiveness  and  suitability. 

7.5,  Managing  Data  During  Test  Conduct.  The  DT&E  test  director  will  establish  data  quality  control  and  accountability 
procedures.  The  RTO  is  responsible  for  acquiring,  monitoring,  controlling,  and  assuring  the  quality  of  test  data.  The  RTO  will 
ensure  the  smooth  flow  of  test  data  to  those  responsible  for  data  processing,  analysis,  and  evaluation. 

7.5.1.  Validating  Data.  The  DT&E  test  team  will  continuously  assess  test  data  to  ensure  it  correctly  represents  the  variables 
it  is  supposed  to  characterize,  and  whether  the  data  adequately  supports  the  objectives  in  the  DT&E  test  plan.  This  data  will 
be  used  to  modify  existing  models,  or  create  new  models  for  pretest  analysis  predictions. 

7.5.2.  Reducing  and  Analyzing  Data.  The  DT&E  test  team  will  sort  and  evaluate  the  data  in  order  to  answer  specific  test 
objectives.  The  process  of  validating  and  analyzing  data  will  be  concurrent  with  test  execution.  The  test  team  will  continually 
determine  if  they  have  associated  the  correct  data  with  the  right  MOEs  and  MOPs,  and  that  planned  analytical  techniques  are 
working.  The  test  director  will  ensure  all  data  reduction  tools  are  available  and  operationally  verified  prior  to  active  testing. 

7.5.3.  Joint  Reliability  and  Maintainability  Evaluation  Team  (JRMET).  The  SM  will  establish  a  JRMET  to  assist  in  the 
collection,  analysis,  verification,  and  categorization  of  R&M  data  during  DT&E  and  OT&E.  A  representative  from  the 
program  office  will  chair  the  JRMET.  The  JRMET  will  include  representatives  from  the  RTO,  PTO,  OTA,  operating  command, 
contractor  (non  voting),  and  other  participating  commands  and  agencies.  The  JRMET  will  establish  a  common  database  from 
which  members  can  independently  evaluate  the  data  according  to  their  needs. 

7.5.4.  Test  Data  Scoring  Board  (TDSB).  The  TDSB  compiles,  reviews,  and  scores  all  available  R&M  data.  The  SM  will 
establish  a  TDSB  and  designate  a  chairperson  from  the  program  office  during  DT&E  and  OT&E.  The  TDSB  is  a 
government-only  group  which  includes  representatives  from  the  RTO,  PTO,  OTA,  operating  command,  and  other  participating 
commands  and  agencies. 

7.5.5.  Disposing  of  Test  Data.  Test  directors  will  maintain  original  data  to  allow  for  additional  analysis,  if  necessary. 
Otherwise,  they  will  dispose  of  data  according  to  AFI  37-138,  Records  Disposition— Procedures  and  Responsibilities. 


AFI 99-101  1  November  1996 


29 


7.5.6.  Pre-Test  and  Post-Test  Briefings.  During  test  conduct,  the  test  director  will  conduct  a  briefing  before  each  test  event 
to  inform  test  team  members  of  previous  results  and  upcoming  tests.  After  the  test  event  is  complete,  test  team  members  will 
debrief  the  test  director  on  test  outcome  and  whether  the  required  data  was  collected. 

7.6.  Monitoring  Tests. 

7.6.1.  Monitoring  Contractor  Testing.  The  SM  and  RTO  will  monitor  testing  conducted  by  contractors.  The  SM,  RTO,  and 
OTA  must  ensure  any  contractor-generated  data  used  for  the  combined  portions  of  DT&E/OT&E  is  protected  from  tampering, 
improper  collection  methods,  manipulation,  bias,  or  conflict  of  interest.  The  greater  the  reliance  on  contractors  for  combined 
test  data,  the  greater  the  need  for  oversight  by  knowledgeable  government  officials.  The  number  of  government  officials  is 
a  management  decision  based  on  a  number  of  factors  such  as  the  technical  complexity  of  the  project  or  its  components,  the 
technical  qualifications  of  government  officials,  and  the  inspection  techniques  available.  Include  monitoring  requirements  in 
appropriate  provisions  of  the  SOW  and  the  contract. 

7.6.2.  Monitoring  Technical  Reviews,  Design  Reviews,  and  Audits.  During  these  events,  the  SM  and  government  testers  will 
examine  the  contractor’s  approach  to  test  problems,  evaluate  the  validity  of  the  test  processes,  and  evaluate  the  accuracy  of 
the  contractor’s  results.  If  government  personnel  find  discrepancies,  they  will  inform  the  SM  for  appropriate  action.  The  SM 
will  ensure  personnel  always  document  observations  and  findings  for  future  reference. 

7.6.3.  Monitoring  Government-Conducted  Testing.  The  SM  will  closely  monitor  government-conducted  testing  and  stay 
involved  with  all  RTO  activities  related  to  the  program.  The  amount  of  SM  involvement  depends  on  the  items  tested,  their 
complexity,  costs,  expected  test  results,  funding,  and  so  forth.  The  RTO  must  develop  procedures  for  conducting  tests  at  a  level 
appropriate  to  the  risk.  The  SM  will  ensure  the  RTO  performs  the  tests  and  analyzes  the  data  using  the  agreed-upon  test 
procedures. 

7.7.  Mishap  Accountability.  Mishap  accountability  will  be  clearly  established  prior  to  conducting  tests.  The  RTO  must 
account  for  aircraft,  drone,  missile,  explosive,  and  ground  mishaps,  nuclear  accidents  and  incidents,  and  safety  deficiencies 
as  specified  in  AFI  91-204,  Investigating  and  Reporting  US  Air  Force  Mishaps,  or  as  established  in  an  MOA  between  the 
involved  agencies.  The  RTO  or  PTO  will  send  a  copy  of  each  MOA  to  HQ  AFMC/DO  no  later  than  10  days  before  beginning 
tests. 


Chapter  8 

TEST  AND  EVALUATION  REPORTING 

8.1.  General  Information.  Test  and  evaluation  reports  play  an  important  role  in  the  decision-making  process  for  weapon 
systems  acquisition.  They  must  be  timely,  factual,  concise,  accurate,  and  complete.  The  SM  and  RTO  will  establish  the 
frequency  and  content  of  reports  early  in  the  test  planning  process.  Test  and  evaluation  reports  will: 

•  Provide  test  results  in  support  of  program  management  decisions. 

•  Assess  the  capabilities  and  limitations  of  weapon  systems  under  development. 

•  Provide  historical  documentation  of  weapon  system  development. 

•  Document  test  techniques,  procedures,  and  data  analysis. 

•  Provide  information  for  Air  Force  Technical  Orders. 

•  Capture  lessons  learned. 

8.2.  Distributing  and  Safeguarding  Test  Reports. 

8.2.1.  Before  Milestone  Decisions.  The  SM  will  provide  DT&E  reports  on  oversight  programs  in  sufficient  time  before  MS 
II  and  III  reviews.  Draft  DT&E  reports  are  due  to  OSD/DTSE&E,  the  PEO,  DAC,  and  HQ  USAF/TE  59  days  before  the  DAB 
review.  Final  DT&E  reports  are  due  to  the  PEO,  DAC,  HQ  USAF/TE,  and  OSD/DTSE&E  33  days  before  DAB  reviews. 
DT&E  briefings  to  Air  Force  principals  may  also  be  directed. 

8.2.2.  SIGINT  and  COMSEC  Reports.  The  SM  will  coordinate  distribution  of  signals  intelligence  (SIGINT)  and  COMSEC 
final  reports  with  NSA,  and  provide  permanent  file  copies  to  HQ  AFIC/IMQF,  San  Antonio  TX  78243-5000. 

8.2.3.  Control  of  Reports.  The  reporting  requirements  in  this  Instruction  are  exempt  from  licensing  according  to  AFI  37-124, 
The  Information  Collections  and  Reports  Management  Program. 

8.3.  Types  of  DT&E  Reports.  DT&E  reports  come  from  contractors  and  government  organizations.  The  SM  documents 
the  requirements  for  contractor  test  reports  in  the  contract.  Contractor  T&E  reports  will  contain  information  similar  to  Air 
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Force-prepared  technical  T&E  reports  (see  AFPAM  99-116,  Test  and  Evaluation  Management  Guide).  Air  Force-developed 
DT&E  reports  are  categorized  as: 

•  Quick  look  reports. 

•  Progress  reports. 

•  Annual  interim  reports. 

•  Preliminary  reports  of  results. 

•  Technical  reports  (or  final  reports). 

•  Technical  letter  reports. 

•  End  of  Test  Phase  report. 

8.4.  End  of  Test  Phase  Report.  Normally,  the  RTO  will  submit  DT&E  reports  at  the  end  of  each  test  phase.  However,  when 
a  program  has  an  extended  test  phase,  the  RTO  will  also  provide  annual  interim  reports.  The  SM  will  send  final  DT&E  reports 
to  the  PEM,  the  OTA,  the  participating  MAJCOMs,  and  DTIC.  For  ACAT I  and  II  programs,  and  programs  on  OSD  oversight, 
the  PEM  will  send  a  copy  to  OSD/DTSE&E  and  OSD/DOT&E.  DT&E  reports  are  not  releasable  to  non-government  agencies 
without  prior  approval  and  coordination  of  the  SM. 

8.5.  Deficiency  Reporting.  Deficiencies  will  be  reported  throughout  the  acquisition  program  as  necessary  to  ensure  proper 
problem  identification,  tracking,  and  corrective  action.  DRs  will  be  reviewed  during  the  certification  process  in  preparation 
for  dedicated  OT&E.  End  of  DT&E  reports  prepared  by  the  SM  will  include: 

•  A  prioritized  list  and  status  of  all  open  category  I  (CAT  I)  DRs,  and  the  top  ten  CAT  II  DRs  in  accordance  with  TO 
00-35D-54. 

•  An  action  plan  for  DR  resolution. 

•  An  assessment  of  how  each  DR  affects  system  operation  and  life  cycle  costs. 

8.6.  Reporting  Lessons  Learned.  The  Air  Force  Lessons  Learned  Program  portion  of  AFAM  gives  SMs,  engineers,  users, 
and  other  acquisition  system  personnel  the  necessary  feedback  on  results  of  past  design  and  acquisition  decisions.  All 
participants  will  record  lessons  learned  as  they  occur.  Contact  the  Acquisition  Model  System  Program  Office  (ASC/CYM, 
DSN  785-0423)  for  procedures  on  how  to  submit  lessons  learned. 

8.7.  Certification  of  System  Readiness  for  Dedicated  OT&E.  The  SM  will  prepare  required  documentation  to  assist  the 
certifying  official  (PEO,  DAC,  or  equivalent  official)  certify  the  system  ready  for  dedicated  OT&E.  The  SM  will  prepare  a 
certification  message  summarizing  the  critical  areas  and  processes  accomplished,  and  discuss  any  agreed-upon  limitations  to 
OT&E.  See  AFMAN  63-119  for  details.  The  certification  message  will  serve  as  a  quantifiable  benchmark  of  projected 
capabilities  against  which  to  check  OT&E  results. 

8.8.  Disposition  of  Test  Assets.  Test  assets  (e.g.,  instrumentation,  data  automation  equipment,  test  articles)  from  canceled 
or  completed  test  programs  will  be  properly  disposed  of  as  follows: 

•  Catalogued  and  turned  over  to  other  RDT&E  programs  or  sustainment  programs. 

•  Refurbished  and  reassigned  to  operational  use  or  inventory. 

•  Turned  over  to  the  nearest  Defense  Reutilization  Management  Office. 


HOWARD  W.  LEAF,  Lt  General,  USAF,  Retired 
Director,  Test  and  Evaluation 
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SEMP-System  Engineering  Management  Plan 
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SRB-Safety  Review  Board 

SRD-System  Requirements  Document 

SSE-systems  security  engineering 

S&T-science  and  technology 

STA-System  Threat  Assessment 

STAR-System  Threat  Assessment  Report 
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Terms 


NOTE:  Joint  Publication  1-02,  Department  of  Defense  Dictionary  of  Military  and  Associated  Terms,  1  Dec  1989,  and  AFM 
11-1,  Air  Force  Glossary  of  Standardized  Terms,  contain  standardized  terms  and  definitions  for  DoD  and  Air  Force  use. 
Acquisition  Category  (ACAT) — Acquisition  categories  determine  the  level  of  review,  decision  authority,  and  applicable 
procedures.  They  facilitate  decentralized  decision  making  and  execution,  and  compliance  with  statutorily  imposed 
requirements.  There  are  three  ACATs  based  on  research,  development,  test,  and  evaluation  (RDT&E)  and/or  procurement  costs 
stated  in  FY  96  dollars: 

•  ACAT  I  programs  are  major  defense  acquisition  programs  (MDAP)  requiring  eventual  expenditure  for  RDT&E  of 
more  than  $355  million,  or  procurement  of  more  than  $2,135  billion. 

•  ACAT  II  programs  are  major  programs  requiring  eventual  expenditure  for  RDT&E  of  $140  million,  or  procurement 
of  more  than  $645  million. 

•  ACAT  III  programs  are  those  programs  not  meeting  the  requirements  for  ACAT  I  or  ACAT  II  programs. 

•  There  are  two  sub-categories  of  ACAT  I  programs: 

•  ACAT  ID  means  the  program  is  subject  to  Defense  Acquisition  Board  (DAB)  oversight,  and  the  Milestone 
Decision  Authority  (MDA)  is  USD(A&T). 

•  ACAT  IC  means  the  MDA  is  the  component  head,  or  Service  Acquisition  Executive  (SAE). 

•  ACAT  lA  programs  are  major  automated  information  systems  (MAIS)  requiring  program  costs  for  any  single  year 
in  excess  of  $30  million,  total  program  costs  in  excess  of  $120  million,  or  total  life-cycle  costs  in  excess  of  $360 
million,  or  those  designated  by  ASD(C3I)  to  be  ACAT  lA, 

•  There  are  two  sub-categories  of  ACAT  I A  programs: 

•  ACAT  lAM  means  the  MDA  is  ASD(C3I). 

•  ACAT  I  AC  means  the  MDA  is  the  component  DAC. 

Acquisition  Community — All  personnel  involved  in  the  conceptualization,  initiation,  design,  development,  test,  contracting, 
production,  deployment,  sustainment,  logistics  support,  modification,  and  disposal  of  weapon  and  other  systems,  supplies,  or 
services  to  satisfy  DoD  needs,  and  intended  for  use  in  or  in  support  of  military  missions. 
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Acquisition  Decision  Memorandum  (ADM) — A  memorandum  signed  by  the  milestone  decision  authority  documenting  the 
decisions  made  and  the  exit  criteria  established  as  the  result  of  a  milestone  decision  review  or  in-process  review,  (DoD 
Acquisition  Deskbook) 

Acquisition  Phases — The  logical  parts  of  an  acquisition  program,  separated  by  milestone  decision  points,  during  which 
broadly  stated  mission  needs  are  progressively  translated  into  well-defined  system-specific  requirements.  The  number  of 
phases  shall  be  tailored  to  meet  the  specific  needs  of  individual  acquisition  programs.  The  four  acquisition  phases  are: 

•  Concept  Exploration. 

•  Program  Definition  and  Risk  Reduction. 

•  Engineering  and  Manufacturing  Development. 

•  Production,  Fielding/Deployment,  and  Operational  Support. 

Acquisition  Process — ^The  system  of  discrete,  logical  phases  separated  by  major  decision  points  called  milestones.  The 
acquisition  process  begins  when  broad  mission  needs  are  identified  which  cannot  be  satisfied  with  non-materiel  solutions.  AFI 
63-101. 

Acquisition  Program  Baseline  (APB) — A  succinct  document  that  details  cost,  schedule,  and  performance  (including  support) 
parameters,  and  program  breach  information.  It  establishes  the  commitment  between  the  program  manager  and  the  Milestone 
Decision  Authority  (AFI  63-101). 

Acquisition  System — K  single  uniform  system  whereby  all  equipment,  facilities,  and  services  are  planned,  designed, 
developed,  tested,  acquired,  maintained,  and  disposed  of  within  the  DoD,  The  system  encompasses  establishing  and  enforcing 
policies  and  practices  that  govern  acquisitions,  to  include  documenting  mission  needs  and  establishing  performance  goals  and 
baselines;  determining  and  prioritizing  resource  requirements  for  acquisition  programs;  planning  and  executing  acquisition 
programs;  directing  and  controlling  the  acquisition  review  process;  developing  and  assessing  logistics  implications; 
contracting;  monitoring  the  execution  status  of  approved  programs;  and  reporting  to  Congress.  See  DoDD  5134.1,  Under 
Secretary  of  Defense  (Acquisition  &  Technology), 

Advanced  Concept  Technology  Demonstration  (ACTD) — A  means  of  rapidly  demonstrating  the  use  of  mature  technologies 
to  address  urgent  military  needs.  ACTDs  are  designed  to  rapidly  transfer  technology  from  developers  to  users.  Demonstrations 
are  jointly  developed  and  implemented  with  the  operational  user  and  development  commun-ities  as  key  participants.  The 
fundamental  goals  are  to  provide  a  sound  basis  for  invest-ment  decisions,  and  provide  residual  operational  capabilities.  ACTDs 
are  funded  by  OSD. 

Automated  Information  System  (AIS) — Combination  of  computer  hardware  and  software,  data,  or  telecommunications,  that 
performs  functions  such  as  collecting,  processing,  transmitting,  and  displaying  information.  Excluded  are  computer  resources, 
both  hardware  and  software,  that  are:  physically  part  of,  dedicated  to,  or  essential  in  real  time  to  the  mission  performance  of 
weapons  systems.  (DoDD  5000,1) 

Availability  (Ao) — A  measure  of  the  degree  to  which  an  item  is  in  the  operable  and  commitable  state  at  the  start  of  a  mission 
when  the  mission  is  called  for  at  an  unknown  (random)  time. 

Combined  Test  Force  (CTF) — An  integrated  T&E  product  team  empowered  to  evaluate  a  weapon  system  by  collocating  its 
major  members  at  one  primary  test  site.  The  requirements,  resources,  test  objectives,  and  leadership  of  various  test  efforts  are 
integrated  to  achieve  higher  levels  of  efficiency.  As  a  minimum,  representatives  from  the  DT&E  and  OT&E  communities, 
contractors,  and  operating  commands  will  be  members. 

Combined  Testing — Testing  conducted  by  the  developmental  and  operational  testers  when  there  are  clear  cost  and/or 
schedule  advantages.  The  high  cost  or  lack  of  sufficient  test  articles  may  provide  an  overall  benefit  for  DT&E  and  OT&E 
teams  to  share  test  resources  and  data.  Combined  testing  usually  ends  with  a  phase  of  dedicated  OT&E. 
Commercial-off-tbe  Shelf.  Any  item  that  is: — 

•  Commercially  available  in  the  marketplace. 

•  Any  previously  developed  item  of  supply  that  is  in  use  by  a  department  or  agency  of  the  United  States,  a  state  or  local 
government,  or  a  foreign  government  with  which  the  United  States  has  a  mutual  defense  cooperation  agreement, 

•  Any  of  the  above  items  that  requires  only  minor  modification  to  meet  the  requirements  of  the  procuring  agency. 

•  Any  of  the  above  items  of  supply  that  is  currently  being  produced  but  is  not  yet  in  use  or  is  not  yet  available  in  the 
commercial  marketplace  (DoD  5000.2-R  and  Title  41  U.S.C.  403). 

•  Any  nondevelopmental  item  (NDI). 

Compatibility — ^The  capability  of  two  or  more  items  or  components  of  equipment  or  material  to  exist  or  function  in  the  same 
system  or  environment  without  mutual  interference  (CJCSI  6212.01A). 

Concurrency — ^The  production  of  a  system  while  developmental  activities  are  still  ongoing.  The  risk  in  concurrency  is  that 
of  producing  a  large  number  of  units,  the  configuration  of  which  might  later  prove  to  be  ineffective  or  unsuitable. 

Covered  System — A  vehicle,  weapon  platform,  or  conventional  weapon  system  that  includes  features  designed  to  provide 
some  degree  of  protection  to  users  in  combat;  and  that  is  a  major  system  (10  U.S.C.  2366).  A  term  used  to  describe  the  systems 
which  must  undergo  LFT&E. 
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Critical  Operational  Issue  (COI) — A  key  question  that  must  be  examined  in  operational  test  and  evaluation  to  determine  the 
system’s  capability  to  perform  its  mission.  Testers  normally  phrase  a  COI  as  a  question  to  be  answered  in  evaluating  a  system’s 
operational  effectiveness  or  suitability. 

Defense  Acquisition  Deskbook — An  automated  repository  of  information  consisting  of  an  electronic  Desk  Reference  Set,  a 
Tool  Catalog,  and  a  Forum  for  the  exchange  of  information.  The  Reference  Set  organizes  information  into  two  main 
categories:  mandatory  guidance;  and  discretionary  information. 

Deficiency — A  condition  that  prevents  successful  mission  accomplishment,  or  degrades  a  system’s  operational  effectiveness 
or  suitability  (TO  00-35D-54). 

Deficiency  Report  (DR) — ^The  report  used  to  identify,  document,  and  track  system  deficiency  and  enhancement  data  while 
a  system  is  in  advanced  development,  T&E,  or  operational  transition  (TO  00-35D-54). 

Designated  Acquisition  Commander  (DAC) — ^The  individual  who  functions  as  the  milestone  decision  authority  (MDA)  on 
programs  not  assigned  to  a  PEO.  The  commanders  of  product  centers  and  air  logistics  centers  act  in  this  capacity.  DACs,  like 
PEOs,  are  accountable  to  the  Service  Acquisition  Executive  (SAE)  (AFPD  63-1). 

Developmental  Test  and  Evaluation  (DT&E) — ^Test  and  evaluation  conducted  to  evaluate  design  approaches,  validate 
analytical  models,  quantify  contract  technical  performance  and  manufacturing  quality,  measure  progress  in  system  engineering 
design  and  development,  minimize  design  risks,  predict  integrated  system  operational  performance  (effectiveness  and 
suitability)  in  the  intended  environment,  and  identify  system  problems  (or  deficiencies)  to  allow  for  early  and  timely  resolution 
or  correction.  DT&E  may  include  contractor  testing  (AFPD  99-1). 

Director,  Test,  Systems  Engineering  and  Evaluation  (DTSE&E) — The  focal  point  in  the  Office  of  the  Secretary  of  Defense 
for  DT&E  and  the  T&E  infrastructure,  including  the  Major  Range  and  Test  Facility  Base. 

Early  Operational  Assessment  (EOA) — An  operational  assessment  conducted  before  or  at  MS  II  (DoD  Acquisition 
Deskbook).  An  EOA  assesses  the  most  promising  design  approach  sufficiently  early  in  the  acquisition  process  to  assure  it  has 
the  potential  to  fulfill  user  requirements.  Also  see  operational  assessment. 

Evaluation  Criteria — Standards  by  which  accomplishments  of  required  technical  and  operational  effectiveness  and/or 
suitability  characteristics  or  resolution  of  operational  issues  may  be  addressed  (DoD  Acquisition  Deskbook). 

Evolutionary  Acquisition  (EA) — An  acquisition  strategy  in  which  a  core  capability  is  fielded,  and  the  system  design  has  a 
modular  structure  and  provisions  for  future  upgrades  and  changes  (increments)  as  requirements  are  refined.  This  strategy  is 
well  suited  to  high  technology  and  software  intensive  programs  where  requirements  beyond  a  core  capability  can  generally, 
but  not  specifically,  be  defined.  The  evolutionary  acquisition  strategy  differs  from  incremental  acquisition  because  the  total 
functional  capability  is  not  completely  defined  at  program  inception,  but  evolves  as  the  system  is  built. 

Exit  Criteria — A  set  of  specific  accomplishments  that  must  be  satisfactorily  demonstrated  before  a  program  progresses 
further  in  the  current  acquisition  phase,  or  transitions  to  the  next  acquisition  phase.  Exit  criteria  may  include  such  factors  as 
critical  test  issues,  the  attainment  of  projected  growth  curves  and  baseline  parameters,  and  the  results  of  risk  reduction  efforts. 
Exit  criteria  supplement  minimum  required  accomplishments  and  are  specific  to  each  acquisition  phase  (DoD  Acquisition 
Deskbook). 

First  Article — ^The  preproduction  models,  initial  production  samples,  test  samples,  first  lots,  pilot  models,  and  pilot  lots. 
First  Article  Test  (FAT) — The  T&E  of  first  production  articles  for  conformance  with  specified  contract  requirements  before 
or  during  the  initial  stage  of  production.  FATs  are  conducted  on  each  item  in  a  production  lot.  They  are  repeated  when  the 
process  or  design  changes  significantly,  or  when  a  different  production  source  is  brought  on  line. 

Foreign  Comparative  Test  (FCT) — A  OSD-funded  program  that  allows  each  Service  to  test  foreign-developed  systems, 
components,  equipment  items,  or  technologies.  The  goal  is  to  determine  if  foreign  items  meet  validated  needs  and 
requirements,  and  if  they  are  viable  candidates  for  a  competitive  acquisition.  (See  AFI  99-114) 

Hardware-in-the-Loop  (HITL) — ^Testing  that  involves  system  or  subsystem  hardware  in  an  open  or  closed-loop  mode 
against  high  fidelity  targets  and  threat  simulations.  It  allows  testers  to  test  developmental  and  production  systems  under 
controllable,  repeatable,  non-destructive  conditions. 

Hazard  Control — Methods  used  to  reduce  the  probability  or  severity  of  potential  mishaps.  The  precedence  of  control  is: 
eliminate  or  reduce  hazard  risk  through  design  change;  incorporate  safety  devices;  incorporate  warning  devices  to  detect  and 
alert  personnel  of  hazardous  conditions;  and,  least  acceptable,  develop  procedures  and  training  to  minimize  the  probability  of 
mishaps. 

Implementing  Command — ^The  lead  command  or  agency  designated  by  the  Service  Acquisition  Executive  to  manage  an 
acquisition  program. 

Incremental  Acquisition — An  acquisition  strategy  which  develops  and  deploys  systems  or  functionality  through  a  number 
of  clearly  defined  increments  that  stand  on  their  own.  The  number,  size,  and  phasing  of  all  increments  must  be  defined  for 
the  total  system.  This  strategy  is  most  appropriate  when  user  requirements  are  well  understood  and  easily  defined,  but  a  phased 
approach  is  more  prudent  or  beneficial. 
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Independent  Verification  and  Validation  (IV &V) — ^The  conduct  of  verification  and  validation  of  a  model  or  simulation  by 
individuals  or  agencies  that  did  not  develop  the  model  or  simulation. 

Industrial  and  Maintenance  Inspection  (IMI) — IMIs  are  sustainment  related  inspections  and  quality  checks  performed  to 
verify  completeness  and  accuracy  of  established  repair  or  production  processes,  including  acceptance  testing  and  failure 
investigations  which  are  not  necessarily  associated  with  production  or  repair.  Note:  MI  activities  are  non-T&E  activities 
which  do  not  follow  the  test  process  guidance. 

Initial  Operational  Test  and  Evaluation  (lOT&E) — ^The  operational  test  and  evaluation  conducted  on  production  or 
production-representative  articles  to  help  decide  whether  to  proceed  beyond  low-rate  initial  production.  lOT&E  is  conducted 
to  estimate  how  well  the  system  attains  operational  effectiveness  and  suitability  (DoD  5000.2-R). 

Integrated  Logistics  Support  (ILS) — A  composite  of  all  support  considerations  necessary  to  ensure  the  effective  and 
economical  support  of  a  system  for  its  life  cycle.  A  disciplined,  unified,  and  iterative  approach  to  the  management  and 
technical  activities  necessary  to: 

•  Integrate  the  needed  level  of  support  into  system  and  equipment  design. 

•  Develop  support  requirements  that  consistently  relate  to  readiness  objectives,  to  design,  and  to  other  support 
requirements. 

•  Get  the  required  support. 

•  Give  the  required  support  during  the  operational  phase  at  a  minimum  cost. 

Integrated  Logistics  Support  Plan  (ILSP) — An  Air  Force  management  plan  for  the  integrated  logistics  support  (ILS) 
process.  This  plan  includes  ILS  elements  that  are  integrated  with  each  other  and  also  with  program  planning,  engineering, 
designing,  testing,  and  evaluation  during  production  and  operation.  It  integrates  support  elements  with  the  mission  elements 
of  a  system  throughout  its  life  cycle.  (DoD  5000.2-R/Air  Force  Supplement  1) 

Integrated  Program  Summary  (IPS) — A  DoD  component  document  prepared  and  submitted  to  the  MDA  at  MS  reviews. 
It  succinctly  states  the  status  of  a  program  and  its  readiness  to  proceed  to  the  next  phase  of  the  acquisition  cycle  (DoD 
5000.2-R). 

Integrated  Weapon  System  Management  (IWSM) — ^A  management  concept  that  integrates  all  life-cycle  management 
activities  of  a  system  under  one  manager.  IWSM  entails  “cradle-to-grave”  or  “seamless”  management.  The  acquisition 
process,  sustainment,  FMS,  and  international  cooperative  activities  are  under  the  IWSM  architecture.  See  AFMC  Pamphlet 
800-60,  Integrated  Weapon  System  Management  Guide. 

Integration — ^The  arrangement  of  systems  in  an  architecture  so  that  they  function  together  in  an  efficient  and  logical  way 
(CJCSI  6212.01A). 

Integration  Laboratories — Test  facilities  that  use  real-time  digital  models  of  electronic  combat  and  avionics  systems  to 
evaluate  software  and  hardware  concepts,  technologies,  components,  systems,  and  entire  avionics  suites  in  a  controlled 
environment. 

Interoperability — The  ability  of  systems,  units,  or  forces  to  provide  services  to  and  accept  services  from  other  systems,  units, 
or  forces  and  to  use  these  services  to  help  them  operate  effectively  together.  The  conditions  achieved  among 
communications-electronics  systems  or  items  of  communications-electronics  equipment  when  information  or  services  can  be 
exchanged  directly  and  satisfactorily  between  them  and/or  their  users  (CJCSI  621 2.01  A). 

Joint  Program — Any  defense  acquisition  system,  subsystem,  component,  or  technology  program  involving  formal 
management  or  funding  by  more  than  one  DoD  component  during  any  phase  of  a  system’s  life  cycle. 

Joint  Reliability  and  Maintainability  Evaluation  Team  (JRMET) — ^The  team  responsible  for  collecting,  analyzing,  and 
categorizing  R&M  data  during  DT&E  and  OT&E.  It  is  chaired  by  the  SPO  and  includes  representatives  from  the  supporting 
and  operating  conunands,  the  DT&E  and  OT&E  test  teams,  and,  when  appropriate,  system  contractor  personnel  as  nonvoting 
members. 

Joint  Test  and  Evaluation  (JT&E) — JT&E  programs  evaluate  technical  or  operational  concepts  which  are  applicable  to  more 
than  one  Service.  JT&E  candidate  programs  are  nominated  by  the  Services,  and  directed  and  funded  by  OSD.  They  never 
result  in  the  acquisition  of  systems. 

Lot  Acceptance  Test  (LAT) — ^The  T&E  of  first  production  articles  for  conformance  with  specified  contract  requirements 
before  or  during  the  initial  stage  of  production.  LATs  are  conducted  on  a  random  sample  taken  from  each  production  lot.  They 
are  repeated  when  the  process  or  design  changes  significantly,  or  when  a  different  production  source  is  brought  on  line. 
Lead  Service — ^The  Service  designated  by  USD(A&T)  to  be  responsible  for  management  of  a  system  acquisition  involving 
two  or  more  DoD  components  in  a  joint  program. 

Lethality — The  ability  of  a  munition  system  (or  laser,  high  power  microwave,  etc.)  to  cause  damage  that  results  in  the  loss 
or  degradation  of  the  ability  of  a  target  system  to  complete  its  designated  mission(s)  (DoD  5000.2-R,  Appendix  IV). 

Life  Cycle  Cost — The  total  cost  to  the  Government  of  acquisition  and  ownership  of  a  system  over  its  useful  life.  It  includes 
the  cost  of  development,  acquisition,  support  and,  where  applicable,  disposal  (DoD  Acquisition  Deskbook). 
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Live  Fire  Test  (LFT) — A  test  within  the  OSD  approved  LFT&E  strategy  involving  the  firing  of  actual  munitions  at  target 
components,  subsystems,  subassemblies,  or  system-level  targets  (which  may  or  may  not  be  configured  for  combat)  to  examine 
personnel  casualty,  vulnerability  and/or  lethality  issues  (DoD  5000.2-R,  Appendix  IV). 

Logistics  Supportability — The  degree  to  which  the  planned  logistics  support  allows  the  system  to  meet  its  availability  and 
wartime  usage  requirements.  Planned  logistics  support  includes  the  following:  test,  measurement,  and  diagnostic  equipment; 
spare  and  repair  parts;  technical  data;  support  facilities;  transportation  requirements;  training;  manpower;  software  (DoD 
Acquisition  Deskbook). 

Logistics  Test  and  Evaluation  (LT&E) — The  test  methodology,  criteria,  and  tools  for  evaluating  and  analyzing  the  ten 
Integrated  Logistics  Support  (ILS)  elements  as  they  apply  to  a  system  under  test.  The  objective  is  to  influence  the  design 
through  applying  the  ILS  elements  as  early  as  possible  in  the  acquisition  cycle.  LOG  TEST  integrates  the  evaluation  and 
analysis  efforts  of  R&M,  human  factors  engineering,  and  logistics  test,  and  is  an  integral  part  of  the  DT&E  report. 
Low-Rate  Initial  Production  (LRIP) — ^The  minimum  number  of  systems  (other  than  ships  and  satellites)  to  provide 
production  representative  articles  for  operational  test  and  evaluation,  to  establish  an  initial  production  base,  and  to  permit  an 
orderly  increase  in  the  production  rate  sufficient  to  reach  full-rate  production  upon  successful  completion  of  operational  testing 
(DoD  Acquisition  Deskbook). 

Maintainability — The  ability  of  an  item  to  be  retained  in  or  restored  to  a  specified  condition  when  maintenance  is  performed 
by  personnel  having  specified  skill  levels,  using  prescribed  procedures  and  resources,  at  each  prescribed  level  of  maintenance 
and  repair  (DoD  Acquisition  Deskbook). 

Major  Automated  Information  System  (MAIS) — An  AIS  acquisition  program  that  is  (1)  designated  by  ASD(C3I)  as  a 
MAIS,  or  (2)  estimated  to  require  program  costs  in  any  single  year  in  excess  of  $30  million  in  FY  1996  constant  dollars,  total 
program  costs  in  excess  of  $120  million  in  FY  1996  constant  dollars,  or  total  life-cycle  costs  in  excess  of  $360  million  constant 
dollars.  MAISs  do  not  include  highly  sensitive  classified  programs  (as  determined  by  SECDEF),  See  DoD  Acquisition 
Deskbook. 

Major  Defense  Acquisition  Program  (MDAP) — An  acquisition  program  that  is  not  a  highly  sensitive  classified  program  (as 
determined  by  SECDEF)  and  that  is:  (1)  designated  by  USD(A&T)  as  an  MDAP;  or  (2)  is  estimated  by  USD(A&T)  to  require 
an  eventual  total  expenditure  for  RDT&E  of  more  than  $355  million  in  FY  1996  constant  dollars,  or  for  procurement,  of  more 
than  $2,135  billion  dollars  in  FY  1996  constant  dollars.  See  DoD  Acquisition  Deskbook  and  10  U.S.C.  2430.  An  ACAT  I 
program. 

Major  System — A  combination  of  elements  that  will  function  together  to  produce  the  capabilities  required  to  fulfill  a  mission 
need.  The  elements  may  include  hardware,  equipment,  software,  or  any  combination  thereof,  but  excludes  construction  or 
other  improvements  to  real  property.  A  system  shall  be  considered  a  major  system  if  it  is  estimated  by  USD(A&T)  to  require 
an  eventual  total  expenditure  for  RDT&E  of  more  than  $75  million  in  FY  1980  constant  dollars  (approximately  $140  million 
in  FY  1996  constant  dollars),  or  for  procurement  of  more  than  $300  million  in  FY  1980  constant  dollars  (approximately  $645 
million  in  FY  1996  constant  dollars).  See  DoD  Acquisition  Deskbook  and  10  U.S.C.  2302(5).  An  ACAT  II  program. 
Measure  of  Effectiveness  (MOE) — A  qualitative  or  quantitative  measure  of  a  system’s  performance  or  a  characteristic  that 
indicates  the  degree  to  which  it  performs  the  task  or  meets  a  requirement  under  specified  conditions.  Where  possible,  MOEs 
should  be  defined  to  measure  operational  capabilities  in  terms  of  engagement  or  battle  outcomes  (AFI  10-601). 

Measure  of  Performance  (MOP) — A  quantitative  measure  of  the  lowest  level  of  physical  performance  (e.g.,  range,  velocity, 
payload).  (AFI  10-601). 

Measurement  Facilities — Test  resources  used  for  exploring  and  evaluating  EW  technologies.  Data  collected  from  these 
resources  include  antenna  patterns,  radar  cross  sections,  and  infrared  and  laser  signatures. 

Milestone  (MS) — ^A  decision  point  separating  the  phases  of  an  acquisition  program  requiring  OSD  or  DoD  component 
program  review  or  both.  Milestones  include  both  Defense  Acquisition  Board  (DAB)  and  equivalent  program  reviews  for  DoD 
components.  The  four  milestones  are: 

•  Approval  to  Conduct  Concept  Studies 

•  Approval  to  Begin  a  New  Acquisition  Program 

•  Approval  to  Enter  Engineering  and  Manufacturing  Development 

•  Production  or  Fielding/Deployment  Approval 

Milestone  Decision  Authority  (MDA) — ^The  individual  designated  according  to  criteria  established  by  USD(A&T),  or  by 
ASD(C3I)  for  AIS  programs,  to  approve  entry  of  an  acquisition  program  into  the  next  phase  (DoD  Acquisition  Deskbook). 
Mission  Need  Statement  (MNS) — A  document  prepared  by  users  to  identify  a  requirement  for  a  materiel  solution  to  satisfy 
a  mission  deficiency  (DoD  5000.2-R  and  AFI  10-601).  The  MNS  is  prepared  according  to  CJCS  MOP  77. 

Modification — A  change  to  a  system  that  is  still  in  production,  A  “major  modification”  is  a  modification  that  in  and  of  itself 
meets  the  criteria  of  an  ACAT  I  or  II,  or  is  designated  as  such  by  the  MDA. 


40 


AFI 99-101  Attachment  1  1  November  1996 


Multiservice  Test  and  Evaluation — ^Test  and  evaluation  (T&E)  conducted  by  two  or  more  DoD  components  for  systems  to 
be  acquired  by  more  than  one  DoD  component,  or  T&E  of  a  DoD  component’s  systems  that  must  be  interoperable  with 
another  DoD  component. 

Nondevelopmental  Item  (NDI).  NDIs  are: — 

•  Any  item  commercially  available  in  the  marketplace. 

•  Any  previously  developed  item  of  that  is  in  use  by  a  department  or  agency  of  the  United  States,  a  state  or  local 
government,  or  a  foreign  government  with  which  the  United  States  has  a  mutual  defense  cooperation  agreement. 

•  Any  of  the  above  items  that  requires  only  minor  modification  to  meet  the  requirements  of  the  procuring  agency. 

•  Any  of  the  above  items  of  supply  that  is  currently  being  produced  but  is  not  yet  in  use  or  is  not  yet  available  in  the 
commercial  marketplace  (DoD  5000.2-R  and  Title  41  U.S.C.  403). 

•  Any  commercial-off-the-shelf  (COTS)  item. 

Objective — 

•  An  operationally  significant  increment  above  the  threshold.  An  objective  value  may  be  the  same  as  the  threshold  when 
an  operationally  significant  increment  above  the  threshold  is  not  identifiable  or  useful  (CJCS  MOP  77  and  AFI 
10-601). 

•  That  value  desired  by  the  user  and  which  the  SM  is  contracting  for  or  otherwise  attempting  to  obtain.  The  objective 
value  could  represent  an  operationally  meaningful,  time  critical,  and  cost  effective  increment  above  the  threshold  for 
each  program  parameter.  Program  objectives  (parameters  and  values)  may  be  refined  based  on  the  results  of  the 
preceding  program  phase(s)  (DoD  5000.2-R). 

Operating  Command — The  command  primarily  operating  (using)  a  system,  subsystem,  or  item  of  equipment.  Generally 
applies  to  those  operational  commands  or  organizations  designated  by  HQ  USAF  to  conduct  or  participate  in  operations  or 
operational  testing  (AFI  10-601).  Often  interchanged  with  the  terms  “user”  or  “using  command.” 

Operational  Assessment  (OA) — An  evaluation  of  operational  effectiveness  and  suitability  made  by  an  independent 
operational  test  activity,  with  user  support  as  required,  on  other  than  production  systems.  Testers  may  conduct  OAs  at  any  time 
using  technology  demonstrators,  prototypes,  mockups,  engineering  development  models,  or  simulations,  but  they  may  not 
substitute  an  OA  for  the  independent  OT&E  necessary  to  support  full  production  decisions  (DoD  Acquisition  Deskbook). 
Operational  Effectiveness — ^The  overall  degree  of  mission  accomplishment  of  a  system  used  by  representative  personnel  in 
the  environment  planned  or  expected  (e.g.,  natural,  electronic,  threat)  for  operational  employment  of  the  system  which 
considers  organization,  doctrine,  tactics,  survivability,  vulnerability,  and  threat  (including  countermeasures  and  initial  nuclear 
weapons  effects,  nuclear,  biological,  and  chemical  [NBC]  contamination  threats)  (DoD  Acquisition  Deskbook). 
Operational  Requirements  Document  (ORD) — 

•  A  formatted  statement  containing  performance  and  related  operational  parameters  for  the  proposed  concept  or  system 
(CJCS  MOP  77). 

•  A  document  prepared  by  the  operating  command  that  describes  quantitative  and  qualitative  performance,  operation, 
and  support  parameters,  characteristics,  and  requirements  for  a  specific  candidate  weapon  system.  Its  mandatory 
attachment  is  the  requirements  correlation  matrix  (RCM)  (AFI  10-601). 

Operational  Suitability — ^The  degree  to  which  a  system  can  be  placed  satisfactorily  in  field  use  with  consideration  given  to 
availability,  compatibility,  transportability,  interoperability,  reliability,  wartime  usage  rates,  maintainability,  safety,  human 
factors,  manpower  supportability,  logistics  supportability,  natural  environmental  effects  and  impacts,  documentation,  and 
training  requirements  (DoD  Acquisition  Deskbook). 

Operational  Test  Agency  (OTA) — 

•  Each  Service  has  one  designated  operational  test  agency:  the  Air  Force  has  the  Air  Force  Operational  Test  and 
Evaluation  Center  (AFOTEC);  the  Navy  has  the  Operational  Test  and  Evaluation  Force  (OPTEVFOR);  the  Army  has 
the  Operational  Test  and  Evaluation  Command  (OPTEC);  and  the  Marine  Corps  has  the  Marine  Corps  Operational  Test 
and  Evaluation  Agency  (MCOTEA). 

•  The  command  or  agency  designated  in  the  PMD  or  other  appropriate  program  directive  as  responsible  for  managing 
the  independent  OT&E  of  a  system. 

Operational  Test  and  Evaluation  (OT&E) — Testing  and  evaluation  conducted  in  as  realistic  an  operational  environment  as 
possible  to  estimate  the  prospective  system’s  operational  effectiveness  and  operational  suitability.  In  addition,  OT&E  provides 
information  on  organization,  personnel  requirements,  doctrine,  and  tactics.  It  may  also  provide  data  to  support  or  verify 
material  in  operating  instructions,  publications,  and  handbooks  (AFM  11-1). 

Operations  Security  (OPSEC) — A  process  of  identifying  critical  information  and  analyzing  friendly  actions  attendant  to 
military  operations  and  other  activities  to: 

•  Identify  those  actions  that  hostile  intelligence  systems  can  observe. 

•  Determine  indicators  hostile  intelligence  systems  might  obtain  that  could  be  interpreted  or  pieced  together  to  derive 
critical  information  in  time  to  be  useful  to  adversaries. 
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•  Select  and  execute  measures  to  eliminate  or  reduce  to  an  acceptable  level  the  vulnerability  of  friendly  actions  to 
exploitation  by  adversaries  (JCS  PUB  1-02). 

Oversight  Program — An  acquisition  program  on  OSD’s  Annual  T&E  Oversight  List  which  is  jointly  published  by  DOT&E 
and  DTSE&E.  Generally,  the  list  includes  ACAT  I  (MDAP)  programs,  ACAT  II  (major  system)  programs,  and  any  other 
program  designated  for  T&E  oversight.  The  master  list  designates  oversight  for  three  types  of  testing:  DT&E;  OT&E;  and 
LFT&E.  These  programs  require  some  additional  documentation,  and  have  additional  review  and  approval  requirements.  See 
DoD  5000.2-R  for  details. 

Participating  Command — A  HQ  USAF-designated  command  or  agency  that  takes  an  active  part  in  developing  a  system,  and 
supports  and  advises  the  SM.  The  supporting  command  is  also  a  participating  command  API  10-601). 

Participating  Test  Organization  (PTO) — A  test  organization  required  to  afford  specific  resources  during  DT&E. 

Phase.  — See  acquisition  phase. 

Preplanned  Product  Improvement  (P3I) — A  phased  acquisition  approach  that  incrementally  satisfies  operational 
requirements  in  order  to  address  the  cost,  risk,  or  relative  time  urgency  of  different  elements  of  the  system  being  developed. 
The  deferred  elements  are  developed  in  parallel  or  subsequent  efforts.  P3I  includes  enhancements  planned  for  ongoing  systems 
that  go  beyond  the  current  performance  envelope. 

Preproduction  Qualification  Test  (PPQT) — ^The  formal  contractual  tests  that  ensure  design  integrity  over  the  specified 
operational  and  environmental  range.  These  tests  usually  use  prototype  or  preproduction  items  manufactured  to  the  proposed 
production  design  specifications  and  drawings.  PPQTs  include  contractual  reliability  and  main-tainability  demonstration  tests 
required  prior  to  production  release. 

Production  Article — ^An  article  that  is  in  final  form,  employs  standard  parts  (or  nonstandard  parts  approved  by  the  agency 
concerned),  and  is  representative  of  final  equipment. 

Production  Qualification  Tests  (PQT) — A  technical  test  conducted  after  MS  III  to  ensure  manufacturing  processes, 
equipment,  and  procedures  are  effective.  These  tests  are  conducted  on  a  number  of  samples  taken  at  random  from  the  first 
production  lot,  and  are  repeated  if  the  manufacturing  process  or  design  is  changed  significantly,  or  when  a  second  source  is 
brought  on  line. 

Program  Management  Directive  (PMD) — The  official  Air  Force  document  used  to  direct  acquisition  or  modification 
responsibilities  to  the  appropriate  MAJCOM,  PEG,  or  DAC  for  a  specific  system  and  subsystem’s  development,  acquisition, 
concept  direction  study,  or  modification.  The  PMD  states  the  program’s  unique  requirements,  goals,  and  objectives,  especially 
those  to  be  met  at  each  acquisition  milestone  or  program  review  (HOI  800-2). 

Prototype — A  model  used  to  evaluate  design,  performance,  and  production  potential  (Joint  Pub  1-02).  The  Air  Force  also  uses 
prototypes  during  development  of  a  technology  or  acquisition  program  for  verification  or  demonstration  of  technical 
feasibility.  Prototypes  may  not  be  representative  of  the  final  production  item. 

QualiBcation  Operational  Test  and  Evaluation  (QOT&E) — The  operational  testing  performed  on  programs  instead  of 
lOT&E  for  which  there  is  no  RDT&E-funded  development  effort. 

QualiHcation  Test  and  Evaluation  (T&E) — ^The  testing  performed  on  systems,  and  on  modifications  to  existing  systems,  for 
which  there  is  no  RDT&E-funded  development  effort  (AFPD  99-1). 

Reliability — ^The  ability  of  a  system  and  its  parts  to  perform  its  mission  without  failure,  degradation,  or  demand  on  the  support 
system  (DoD  Acquisition  Deskbook). 

Requirement — ^The  validated  need  of  an  operational  user.  Initially  expressed  in  broad  operational  capability  terms  in  the 
format  of  a  MNS.  It  progressively  evolves  to  system-specific  performance  requirements  in  the  ORD  (CJCS  MOP  77). 
Requirements  Correlation  Matrix  (RCM) — ^A  three-part  matrix  spreadsheet  used  to  provide  an  audit  trail  of  system 
capabilities  and  characteristics  identified  in  the  ORD.  It  lists  thresholds  and  objectives;  identifies  user  recommended  key 
performance  parameters;  provides  supporting  rationale  justifying  each  threshold;  and  preserves  rationale  for  changes  in 
requirements  as  the  system  matures  (API  10-601). 

Research,  Development,  Test  and  Evaluation  (RDT&E) — 

•  The  type  of  funding  appropriation  (3600)  intended  for  research,  development,  test  and  evaluation  efforts.  See  DoD 
7000. 14-R,  Vol  2A,  Financial  Management  Regulation,  and  AFI  65-601,  Vol  I,  Budget  Guidance  and  Procedures. 

•  The  term  “research  and  development”  (R&D)  broadly  covers  the  work  performed  by  a  government  agency  or  the 
private  sector.  “Research”  is  the  systematic  study  directed  toward  fuller  scientific  knowledge  or  understanding  of  a 
subject  area.  “Development”  is  the  systematic  use  of  the  knowledge  and  understanding  gained  from  research  for  the 
production  of  useful  materials,  devices,  systems,  or  methods.  R&D  includes  the  suppor-ting  test  and  evaluation 
activities,  to  include  lOT&E. 

Responsible  Test  Organization  (RTO) — The  lead  government  entity  that  is  qualified  and  responsible  for  DT&E. 

Risk. — A  subjective  assessment  made  regarding  the  likelihood  or  probability  of  not  achieving  a  specific  objective  by  the  time 
established  with  the  resources  provided  or  requested.  It  also  refers  to  overall  program  risk  (DoD  Acquisition  Deskbook). 
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Single-Face-to-Customer  (SFTC)  Office — One  of  three  single  points  of  contact  providing  comprehensive  up-front  planning 
assistance  to  test  organizations.  Organized  around  mission  areas  rather  than  the  test  centers,  these  offices  facilitate  early  T&E 
planning  for  new  programs,  major  modifications,  P3Is,  and  T&E  investment  planning.  The  three  SFTCs  are: 
Airframe-Propulsion/Avionics-Electronic  Warfare;  Space;  and  Armament/Munition s-C^^L 

Single  Manager  (SM) — A  government  official  (military  or  civilian)  who  is  responsible  and  accountable  for  decisions  and 
overall  management  (to  include  all  cost,  schedule,  performance,  and  sustainment)  of  a  system,  product  group,  or  materiel 
group.  Also  known  as  system  program  director,  program  manager,  product  group  manager,  or  materiel  group  manager  (AFMC 
Pamphlet  800-60). 

Specification — ^There  are  many  types  of  “specifications.”  A  document  intended  primarily  for  use  in  procurement  which  clearly 
and  accurately  describes  the  essential  technical  requirements  for  items,  materials,  or  services,  including  the  procedures  by 
which  it  will  be  determined  that  the  requirements  have  been  met.  Also  called  military  specifications. 

Specification,  Performance-Based — Under  acquisition  reform,  new  systems  must  be  described  in  performance  terms,  letting 
bidders  propose  the  “how  to”  details.  Performance-based  specifications  describe  only  “what”  is  needed  and  eliminate  “how 
to”  information.  Only  form,  fit,  and  function  descriptions  are  given;  detailed  configuration  management  controls  and 
production  baseline  descriptions  are  left  to  the  contractor.  Also  known  as  commercial  item  descriptions  or  nongovernment 
standards. 

Supporting  Command — The  command  (usually  Air  Force  Materiel  Command)  responsible  for  providing  logistics  support 
for  a  system  (AFI  21-102). 

Survivability — The  capability  of  a  system  and  crew  to  avoid  or  withstand  a  man-made  hostile  environment  without  suffering 
an  abortive  impairment  of  its  ability  to  accomplish  its  designated  mission.  Its  components  are  susceptibility  and  vulnerability. 
Susceptibility — ^The  degree  to  which  a  weapon  system  is  open  to  effective  attack  due  to  one  or  more  inherent  weaknesses. 
(Susceptibility  is  a  function  of  operational  tactics,  countermeasures,  probability  of  enemy  fielding  a  threat,  etc.)  Susceptibility 
is  considered  a  subset  of  survivability  (DoD  5000.2-R,  Appendix  III). 

Sustainment — Activities  that  sustain  systems  during  the  operations  and  support  phases  of  the  system  life  cycle.  Sustainment 
activities  include  any  investigative  test  and  evaluation  (T&E)  which  extends  the  useful  military  life  of  systems,  or  expands 
the  current  performance  envelope  or  capabilities  of  fielded  systems.  Sustainment  activities  also  include  T&E  for  modifications 
and  upgrade  programs,  and  may  disclose  system  or  product  deficiencies  and  enhancements  that  make  further  acquisitions 
necessary.  The  T&E  conducted  during  sustainment  follows  the  same  guidance  in  this  AFI  as  for  the  DT&E  conducted  during 
the  acquisition  process. 

System  Engineering  Management  Plan  (SEMP) — A  concise,  top-level  management  plan  for  the  integration  of  all  system 
design  activities.  It  forms  the  basis  for  comprehensive  T&E  planning.  The  SEMP  defines  and  describes  the  type  and  degree 
of  system  engineering  management,  the  system  engineering  process,  and  the  integration  of  related  engineering  programs. 
System  Maturity  Matrix  (SMM) — An  acquisition  management  tool  used  to  aid  management  in  tracking  a  program’s 
technical  progress  and  risks.  The  SMM  links  user  requirements  and  system  specifications  with  anticipated  T&E  results.  It 
provides  a  metric  for  program  monitoring  and  reporting  so  true  progress  toward  verification  of  capabilities  and  requirements 
can  be  assessed.  The  SMM  is  coordinated  with  the  user  and  OTA,  and  approved  by  the  PEO  or  DAC.  The  SMM  is  not  a 
substitute  for  a  valid  requirements  document. 

Threshold. — 

•  The  minimum  acceptable  operational  value  for  a  system  capability  or  characteristic  below  which  the  utility  of  the 
system  becomes  questionable.  A  minimum  acceptable  operational  value  for  a  system  capability  or  characteristic  which, 
in  the  user’s  judgment,  is  necessary  to  provide  an  operational  capability  that  will  satisfy  the  mission  need  (CJCS  MOP 
77  and  AFI  10-601). 

•  The  minimum  acceptable  value  which,  in  the  user’s  judgment,  is  necessary  to  satisfy  the  need.  If  threshold  values  are 
not  achieved,  program  performance  is  seriously  degraded,  the  program  may  be  too  costly,  or  the  program  may  no 
longer  be  timely.  The  spread  between  objective  and  threshold  values  shall  be  individually  set  for  each  program  based 
on  the  characteristics  of  the  program  (e.g.,  maturity,  risk)  (DoD  5000.2-R). 

Upgrade — A  change  to  a  system  that  is  out  of  production. 

User — See  operating  command. 

Using  Command — See  operating  command. 

Verification,  Validation  &  Accreditation  (VV&A) — (1)  Verification:  The  process  of  determining  that  a  model 
implementation  accurately  represents  the  developer’s  conceptual  description  and  specifications,  (2)  Validation:  The  process  of 
determining  (a)  the  manner  and  degree  to  which  a  model  is  an  accurate  representation  of  the  real-world  from  the  perspective 
of  the  intended  uses  of  the  model,  and  (b)  the  confidence  that  should  be  placed  on  this  assessment.  (3)  Accreditation:  The 
official  certification  by  users  that  a  model  or  simulation  is  acceptable  for  use  for  a  specific  purpose.  VV&A  is  a  continuous 
process  in  the  life  cycle  of  a  model  as  the  model  gets  upgraded  or  is  used  for  different  applications. 
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Vulnerability — The  characteristics  of  a  system  which  causes  it  to  suffer  a  definite  degradation  (loss  or  reduction  of  capability 
to  perform  its  designated  mission)  as  a  result  of  having  been  subjected  to  a  certain  (defined)  level  of  effects  in  an  unnatural 
(man-made)  hostile  environment.  Vulnerability  is  considered  a  subset  of  survivability. 


